2024-09-18T10:16:36.514 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T10:16:36.527 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T10:16:36.604 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910869 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: skanta@redhat.com first_in_suite: false job_id: '7910869' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T10:16:36.604 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T10:16:36.605 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T10:16:36.605 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T10:16:36.607 INFO:teuthology.task.internal:Checking packages... 2024-09-18T10:16:36.632 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T10:16:36.632 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T10:16:36.632 INFO:teuthology.packaging:ref: None 2024-09-18T10:16:36.632 INFO:teuthology.packaging:tag: None 2024-09-18T10:16:36.632 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T10:16:36.632 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:16:36.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T10:16:36.774 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:16:36.775 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T10:16:36.781 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T10:16:36.781 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T10:16:36.799 INFO:teuthology.task.internal:Saving configuration 2024-09-18T10:16:36.808 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T10:16:36.813 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T10:16:36.836 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 10:10:50.751539', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T10:16:36.836 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T10:16:36.842 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-18T10:16:36.842 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T10:16:36.873 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6df2c9bbe0>, signals=[15]) 2024-09-18T10:16:36.873 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T10:16:36.878 INFO:teuthology.task.internal:Opening connections... 2024-09-18T10:16:36.879 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T10:16:36.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:16:36.971 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T10:16:36.977 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-18T10:16:37.091 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-18T10:16:37.091 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-18T10:16:37.096 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Ubuntu" 2024-09-18T10:16:37.096 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:ID=ubuntu 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE=debian 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="20.04" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_CODENAME=focal 2024-09-18T10:16:37.097 INFO:teuthology.orchestra.run.smithi174.stdout:UBUNTU_CODENAME=focal 2024-09-18T10:16:37.098 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-18T10:16:37.114 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T10:16:37.121 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T10:16:37.127 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T10:16:37.127 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-18T10:16:37.133 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T10:16:37.138 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T10:16:37.138 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-18T10:16:37.153 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T10:16:37.182 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T10:16:37.197 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T10:16:37.197 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T10:16:37.197 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T10:16:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:16:37.204 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-18T10:16:37.211 INFO:teuthology.orchestra.run.smithi174.stdout:5.4.0-124-generic 2024-09-18T10:16:37.211 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.4.0-124-generic 2024-09-18T10:16:37.212 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-09-18T10:16:37.347 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-18T10:16:37.676 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T10:16:37.676 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T10:16:37.730 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T10:16:38.084 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T10:16:38.155 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T10:16:38.303 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T10:16:38.506 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T10:16:38.633 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,598 kB] 2024-09-18T10:16:38.698 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T10:16:38.712 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T10:16:38.713 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T10:16:38.714 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T10:16:38.807 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T10:16:38.814 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T10:16:38.820 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T10:16:38.820 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T10:16:38.828 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T10:16:38.828 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T10:16:38.835 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T10:16:38.864 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T10:16:38.871 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T10:16:38.871 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T10:16:38.871 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T10:16:38.872 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T10:16:38.923 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T10:16:38.923 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T10:16:38.937 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [808 kB] 2024-09-18T10:16:38.945 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T10:16:38.968 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T10:16:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T10:16:39.001 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T10:16:39.002 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T10:16:39.002 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T10:16:39.061 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T10:16:39.062 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T10:16:39.062 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T10:16:39.062 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T10:16:39.063 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T10:16:39.143 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T10:16:39.143 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T10:16:39.144 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T10:16:42.964 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 22.0 MB in 4s (6,196 kB/s) 2024-09-18T10:16:44.448 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T10:16:44.474 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T10:16:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T10:16:44.760 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-18T10:16:44.761 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-18T10:16:44.958 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-09-18T10:16:44.958 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T10:16:44.958 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T10:16:44.959 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T10:16:44.960 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-09-18T10:16:44.960 INFO:teuthology.orchestra.run.smithi174.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T10:16:44.960 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-09-18T10:16:44.960 INFO:teuthology.orchestra.run.smithi174.stdout: thermald 2024-09-18T10:16:45.020 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-09-18T10:16:45.020 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T10:16:45.020 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T10:16:45.020 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T10:16:45.022 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-09-18T10:16:45.022 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T10:16:45.204 INFO:teuthology.orchestra.run.smithi174.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T10:16:45.204 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 77.2 MB of archives. 2024-09-18T10:16:45.205 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T10:16:45.205 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T10:16:46.341 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T10:16:46.614 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T10:16:47.925 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T10:16:47.926 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T10:16:47.926 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T10:16:48.201 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T10:16:48.266 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T10:16:49.004 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 77.2 MB in 3s (23.8 MB/s) 2024-09-18T10:16:49.179 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T10:16:51.017 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T10:16:51.022 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T10:16:51.142 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:16:53.133 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T10:16:53.146 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T10:16:53.262 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:16:54.386 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T10:16:54.400 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T10:16:54.444 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:16:59.257 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T10:16:59.381 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T10:16:59.734 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T10:16:59.860 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T10:17:00.213 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T10:17:00.228 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T10:17:00.296 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T10:17:06.477 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T10:17:06.498 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T10:17:06.543 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:17:09.287 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T10:17:09.442 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T10:17:09.794 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T10:17:09.946 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:17:10.072 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T10:17:10.223 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:17:11.483 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:17:13.476 INFO:teuthology.orchestra.run.smithi174.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T10:17:13.476 INFO:teuthology.orchestra.run.smithi174.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T10:17:13.610 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:17:14.828 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T10:17:14.971 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T10:17:15.131 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T10:17:15.201 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T10:17:15.201 INFO:teuthology.orchestra.run.smithi174.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T10:17:37.132 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T10:17:37.132 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub' 2024-09-18T10:17:37.141 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T10:17:37.154 INFO:teuthology.orchestra.run.smithi174.stdout:Generating grub configuration file ... 2024-09-18T10:17:37.480 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T10:17:37.503 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T10:17:37.917 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T10:17:37.929 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T10:17:40.578 INFO:teuthology.orchestra.run.smithi174.stdout:done 2024-09-18T10:17:41.051 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-09-18T10:17:41.101 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-09-18T10:17:41.101 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-09-18T10:17:41.101 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 20 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.4.0.196.194 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-09-18T10:17:41.102 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-09-18T10:17:41.103 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T10:17:41.103 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T10:17:41.103 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T10:17:41.103 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T10:17:41.103 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T10:17:41.103 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-09-18T10:17:41.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-18T10:17:41.311 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T10:17:41.428 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T10:17:41.510 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T10:17:41.597 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T10:17:43.167 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T10:17:43.196 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T10:17:43.265 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T10:17:43.484 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-18T10:17:43.485 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-18T10:17:43.728 INFO:teuthology.orchestra.run.smithi174.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T10:17:43.728 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T10:17:43.730 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-09-18T10:17:43.750 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-09-18T10:17:43.750 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-09-18T10:17:43.750 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-09-18T10:17:43.750 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-09-18T10:17:43.750 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 20 2024-09-18T10:17:43.750 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.4.0.196.194 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-09-18T10:17:43.751 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-09-18T10:17:43.752 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-18T10:17:43.758 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.6Mv8uu1abK 2024-09-18T10:17:43.758 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub/grub.cfg /tmp/tmp.6Mv8uu1abK 2024-09-18T10:17:43.815 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.6Mv8uu1abK 2024-09-18T10:17:43.855 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.6Mv8uu1abK is 9KB 2024-09-18T10:17:43.869 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.6Mv8uu1abK 2024-09-18T10:17:43.875 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T10:17:43.935 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:17:43.935 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T10:17:43.935 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T10:17:43.971 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T10:17:43.971 DEBUG:teuthology.orchestra.run.smithi174:> sudo update-grub 2024-09-18T10:17:45.334 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub' 2024-09-18T10:17:45.339 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T10:17:45.355 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-09-18T10:17:45.633 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T10:17:45.645 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T10:17:45.993 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T10:17:46.004 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T10:17:47.694 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-09-18T10:17:47.695 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-09-18T10:18:17.729 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T10:18:17.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:18:17.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:18:36.225 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-18T10:18:45.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:18:45.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:18:48.317 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-18T10:19:00.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:19:00.330 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:19:18.721 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-18T10:19:33.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:19:33.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:19:33.978 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-18T10:19:34.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:19:34.495 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T10:19:34.495 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-18T10:19:34.502 INFO:teuthology.orchestra.run.smithi174.stdout:5.4.0-196-generic 2024-09-18T10:19:34.502 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T10:19:34.502 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T10:19:34.502 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T10:19:35.503 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T10:19:35.503 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T10:19:35.607 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-09-18T10:19:35.640 DEBUG:teuthology.parallel:result is None 2024-09-18T10:19:35.641 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T10:19:35.648 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T10:19:35.648 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T10:19:35.655 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T10:19:35.669 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T10:19:35.676 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T10:19:35.677 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T10:19:35.715 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T10:19:35.722 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T10:19:35.722 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T10:19:35.740 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:19:35.750 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:19:35.752 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T10:19:35.758 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T10:19:35.758 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T10:19:35.771 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T10:19:35.833 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T10:19:35.833 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T10:19:35.840 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T10:19:35.847 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T10:19:35.853 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:19:35.854 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T10:19:35.879 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-18T10:19:35.921 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T10:19:35.928 INFO:teuthology.task.internal:Starting timer... 2024-09-18T10:19:35.928 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T10:19:35.939 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T10:19:35.947 DEBUG:teuthology.task.selinux:Excluding smithi174: OS 'ubuntu' does not support SELinux 2024-09-18T10:19:35.947 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T10:19:35.947 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T10:19:35.947 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T10:19:35.947 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T10:19:35.969 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-18T10:19:35.971 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-18T10:19:36.077 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T10:19:36.087 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T10:19:36.088 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T10:25:06.024 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-09-18T10:25:06.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:25:06.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T10:25:06.118 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-18T10:25:06.248 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-18T10:25:06.248 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T10:25:06.259 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T10:25:06.259 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T10:25:06.259 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T10:25:06.292 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T10:25:06.292 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Command line: ntpd -gq 2024-09-18T10:25:06.293 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: proto: precision = 0.100 usec (-23) 2024-09-18T10:25:06.293 INFO:teuthology.orchestra.run.smithi174.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T10:25:06.293 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T10:25:06.293 INFO:teuthology.orchestra.run.smithi174.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T10:25:06.294 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T10:25:06.294 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T10:25:06.295 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T10:25:06.295 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T10:25:06.295 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listen normally on 3 enp3s0f1 172.21.15.174:123 2024-09-18T10:25:06.295 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listen normally on 4 lo [::1]:123 2024-09-18T10:25:06.295 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:771b%5]:123 2024-09-18T10:25:06.295 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:06 ntpd[13683]: Listening on routing socket on fd #22 for interface updates 2024-09-18T10:25:07.292 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:07.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:07.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:07.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:07.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:07.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:07.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:07.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:07.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:07.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:07.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:07.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:07 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:09.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:09.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:09 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:11.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:11.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:11.296 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:11.296 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:11 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:13.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:13.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:13.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:13.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:13.293 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stdout:18 Sep 10:25:13 ntpd[13683]: ntpd: time slew +0.002270 s 2024-09-18T10:25:13.294 INFO:teuthology.orchestra.run.smithi174.stdout:ntpd: time slew +0.002270s 2024-09-18T10:25:13.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T10:25:13.295 INFO:teuthology.orchestra.run.smithi174.stderr:18 Sep 10:25:13 ntpd[13683]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T10:25:13.370 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T10:25:13.370 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-09-18T10:25:13.371 INFO:teuthology.orchestra.run.smithi174.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:25:13.371 INFO:teuthology.orchestra.run.smithi174.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:25:13.371 INFO:teuthology.orchestra.run.smithi174.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:25:13.371 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T10:25:13.372 INFO:teuthology.run_tasks:Running task install... 2024-09-18T10:25:13.386 DEBUG:teuthology.task.install:project ceph 2024-09-18T10:25:13.386 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T10:25:13.387 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T10:25:13.387 INFO:teuthology.task.install:Using flavor: default 2024-09-18T10:25:13.395 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T10:25:13.395 INFO:teuthology.task.install:extra packages: [] 2024-09-18T10:25:13.395 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-key list | grep Ceph 2024-09-18T10:25:13.498 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T10:25:13.558 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T10:25:13.559 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T10:25:13.560 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T10:25:13.560 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:25:13.729 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T10:25:13.730 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:25:13.866 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:25:13.866 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T10:25:13.882 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-09-18T10:25:14.081 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T10:25:14.081 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T10:25:14.092 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T10:25:14.114 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T10:25:14.143 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T10:25:14.168 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T10:25:14.196 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T10:25:14.228 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T10:25:14.264 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T10:25:14.724 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 28.4 kB in 1s (45.2 kB/s) 2024-09-18T10:25:15.884 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T10:25:15.912 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:25:15.982 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-09-18T10:25:16.198 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-09-18T10:25:16.198 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-09-18T10:25:16.392 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T10:25:16.393 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T10:25:16.393 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T10:25:16.394 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-09-18T10:25:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T10:25:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T10:25:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T10:25:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T10:25:16.394 INFO:teuthology.orchestra.run.smithi174.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T10:25:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T10:25:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T10:25:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T10:25:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T10:25:16.395 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T10:25:16.396 INFO:teuthology.orchestra.run.smithi174.stdout: socat xmlstarlet 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T10:25:16.397 INFO:teuthology.orchestra.run.smithi174.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T10:25:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T10:25:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T10:25:16.398 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-09-18T10:25:16.398 INFO:teuthology.orchestra.run.smithi174.stdout: btrfs-tools 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T10:25:16.518 INFO:teuthology.orchestra.run.smithi174.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T10:25:16.520 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T10:25:16.521 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-09-18T10:25:16.522 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 librbd1 2024-09-18T10:25:16.591 INFO:teuthology.orchestra.run.smithi174.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T10:25:16.591 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 201 MB of archives. 2024-09-18T10:25:16.591 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T10:25:16.591 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T10:25:16.689 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T10:25:16.691 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T10:25:16.693 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T10:25:16.705 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T10:25:16.712 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T10:25:16.740 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T10:25:16.742 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T10:25:16.749 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T10:25:16.787 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T10:25:16.789 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T10:25:16.796 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T10:25:16.798 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T10:25:16.799 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T10:25:16.801 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T10:25:16.801 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T10:25:16.801 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T10:25:16.802 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T10:25:16.809 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T10:25:16.825 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T10:25:16.825 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T10:25:16.825 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T10:25:16.826 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T10:25:16.827 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T10:25:16.828 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T10:25:16.828 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T10:25:16.830 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T10:25:16.830 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T10:25:16.852 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T10:25:16.854 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T10:25:16.854 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T10:25:16.862 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T10:25:16.863 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T10:25:16.864 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T10:25:16.865 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T10:25:16.874 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T10:25:16.876 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T10:25:16.877 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T10:25:16.895 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T10:25:16.897 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T10:25:16.898 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T10:25:16.900 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T10:25:16.900 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T10:25:16.901 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T10:25:16.902 INFO:teuthology.orchestra.run.smithi174.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T10:25:16.961 INFO:teuthology.orchestra.run.smithi174.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T10:25:16.969 INFO:teuthology.orchestra.run.smithi174.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T10:25:16.976 INFO:teuthology.orchestra.run.smithi174.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T10:25:16.977 INFO:teuthology.orchestra.run.smithi174.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T10:25:16.985 INFO:teuthology.orchestra.run.smithi174.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T10:25:16.985 INFO:teuthology.orchestra.run.smithi174.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T10:25:16.989 INFO:teuthology.orchestra.run.smithi174.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T10:25:16.998 INFO:teuthology.orchestra.run.smithi174.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T10:25:17.016 INFO:teuthology.orchestra.run.smithi174.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T10:25:17.017 INFO:teuthology.orchestra.run.smithi174.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T10:25:17.036 INFO:teuthology.orchestra.run.smithi174.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T10:25:17.036 INFO:teuthology.orchestra.run.smithi174.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T10:25:17.037 INFO:teuthology.orchestra.run.smithi174.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T10:25:17.103 INFO:teuthology.orchestra.run.smithi174.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T10:25:17.103 INFO:teuthology.orchestra.run.smithi174.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T10:25:17.104 INFO:teuthology.orchestra.run.smithi174.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T10:25:17.105 INFO:teuthology.orchestra.run.smithi174.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T10:25:17.106 INFO:teuthology.orchestra.run.smithi174.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T10:25:17.106 INFO:teuthology.orchestra.run.smithi174.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T10:25:17.108 INFO:teuthology.orchestra.run.smithi174.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T10:25:17.109 INFO:teuthology.orchestra.run.smithi174.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T10:25:17.110 INFO:teuthology.orchestra.run.smithi174.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T10:25:17.113 INFO:teuthology.orchestra.run.smithi174.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T10:25:17.115 INFO:teuthology.orchestra.run.smithi174.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T10:25:17.124 INFO:teuthology.orchestra.run.smithi174.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T10:25:17.139 INFO:teuthology.orchestra.run.smithi174.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T10:25:17.140 INFO:teuthology.orchestra.run.smithi174.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T10:25:17.145 INFO:teuthology.orchestra.run.smithi174.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T10:25:17.150 INFO:teuthology.orchestra.run.smithi174.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T10:25:17.166 INFO:teuthology.orchestra.run.smithi174.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T10:25:17.169 INFO:teuthology.orchestra.run.smithi174.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T10:25:17.220 INFO:teuthology.orchestra.run.smithi174.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T10:25:17.221 INFO:teuthology.orchestra.run.smithi174.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T10:25:17.221 INFO:teuthology.orchestra.run.smithi174.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T10:25:17.221 INFO:teuthology.orchestra.run.smithi174.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T10:25:17.222 INFO:teuthology.orchestra.run.smithi174.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T10:25:17.222 INFO:teuthology.orchestra.run.smithi174.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T10:25:17.240 INFO:teuthology.orchestra.run.smithi174.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T10:25:17.240 INFO:teuthology.orchestra.run.smithi174.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T10:25:17.240 INFO:teuthology.orchestra.run.smithi174.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T10:25:17.244 INFO:teuthology.orchestra.run.smithi174.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T10:25:17.266 INFO:teuthology.orchestra.run.smithi174.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T10:25:17.267 INFO:teuthology.orchestra.run.smithi174.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T10:25:17.267 INFO:teuthology.orchestra.run.smithi174.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T10:25:17.270 INFO:teuthology.orchestra.run.smithi174.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T10:25:17.271 INFO:teuthology.orchestra.run.smithi174.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T10:25:17.658 INFO:teuthology.orchestra.run.smithi174.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T10:25:17.708 INFO:teuthology.orchestra.run.smithi174.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T10:25:17.712 INFO:teuthology.orchestra.run.smithi174.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T10:25:17.713 INFO:teuthology.orchestra.run.smithi174.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T10:25:17.732 INFO:teuthology.orchestra.run.smithi174.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T10:25:17.810 INFO:teuthology.orchestra.run.smithi174.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T10:25:18.258 INFO:teuthology.orchestra.run.smithi174.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T10:25:18.258 INFO:teuthology.orchestra.run.smithi174.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T10:25:18.270 INFO:teuthology.orchestra.run.smithi174.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T10:25:18.289 INFO:teuthology.orchestra.run.smithi174.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T10:25:18.289 INFO:teuthology.orchestra.run.smithi174.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T10:25:18.290 INFO:teuthology.orchestra.run.smithi174.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T10:25:18.387 INFO:teuthology.orchestra.run.smithi174.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T10:25:18.518 INFO:teuthology.orchestra.run.smithi174.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T10:25:18.518 INFO:teuthology.orchestra.run.smithi174.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T10:25:19.616 INFO:teuthology.orchestra.run.smithi174.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T10:25:19.620 INFO:teuthology.orchestra.run.smithi174.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T10:25:19.621 INFO:teuthology.orchestra.run.smithi174.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T10:25:19.939 INFO:teuthology.orchestra.run.smithi174.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T10:25:20.576 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 201 MB in 3s (58.9 MB/s) 2024-09-18T10:25:20.748 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T10:25:20.805 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T10:25:20.809 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T10:25:20.896 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T10:25:21.232 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:21.358 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T10:25:22.039 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:22.163 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T10:25:22.841 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T10:25:22.861 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:22.899 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:23.260 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rados. 2024-09-18T10:25:23.276 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:23.318 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:23.654 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T10:25:23.671 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:25:23.712 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:23.948 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T10:25:23.964 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:24.006 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:24.342 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T10:25:24.361 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:25:24.401 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:24.720 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T10:25:24.735 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T10:25:24.786 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T10:25:25.047 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T10:25:25.063 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:25.097 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:25.450 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T10:25:25.465 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T10:25:25.508 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T10:25:25.894 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T10:25:25.910 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T10:25:25.952 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T10:25:26.288 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T10:25:26.305 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T10:25:26.347 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T10:25:26.733 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T10:25:26.751 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T10:25:26.791 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T10:25:27.119 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T10:25:27.135 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T10:25:27.177 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T10:25:27.672 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T10:25:27.688 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:25:27.741 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:25:28.241 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T10:25:28.259 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:25:28.300 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:25:28.661 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T10:25:28.678 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T10:25:28.719 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:25:29.131 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T10:25:29.150 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T10:25:29.188 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T10:25:29.541 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librgw2. 2024-09-18T10:25:29.557 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:29.599 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:30.352 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T10:25:30.367 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:30.410 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:30.779 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liboath0. 2024-09-18T10:25:30.798 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T10:25:30.838 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T10:25:31.149 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T10:25:31.164 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:31.207 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:31.685 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-common. 2024-09-18T10:25:31.703 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:31.743 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:34.298 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-base. 2024-09-18T10:25:34.315 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:34.367 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:35.277 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T10:25:35.293 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T10:25:35.335 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T10:25:35.722 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T10:25:35.741 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T10:25:35.780 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T10:25:36.099 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T10:25:36.115 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T10:25:36.158 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T10:25:36.427 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T10:25:36.446 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:25:36.485 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:36.865 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T10:25:36.880 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T10:25:36.923 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T10:25:37.251 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T10:25:37.267 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T10:25:37.309 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T10:25:37.662 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T10:25:37.678 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T10:25:37.720 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T10:25:38.048 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T10:25:38.064 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T10:25:38.098 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T10:25:38.409 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T10:25:38.424 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T10:25:38.467 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T10:25:38.770 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T10:25:38.785 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T10:25:38.853 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T10:25:39.173 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-mako. 2024-09-18T10:25:39.191 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T10:25:39.231 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T10:25:39.542 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T10:25:39.559 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T10:25:39.600 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T10:25:39.894 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T10:25:39.914 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T10:25:39.952 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T10:25:40.264 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webob. 2024-09-18T10:25:40.280 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T10:25:40.322 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T10:25:40.666 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T10:25:40.684 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T10:25:40.724 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T10:25:41.010 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T10:25:41.030 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T10:25:41.069 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T10:25:41.427 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T10:25:41.446 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T10:25:41.488 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T10:25:41.813 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T10:25:41.829 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T10:25:41.871 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T10:25:42.208 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-paste. 2024-09-18T10:25:42.224 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T10:25:42.266 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T10:25:42.552 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T10:25:42.569 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T10:25:42.610 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T10:25:42.946 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T10:25:42.965 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T10:25:43.004 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T10:25:43.274 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T10:25:43.291 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T10:25:43.332 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T10:25:43.643 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T10:25:43.660 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T10:25:43.701 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T10:25:44.093 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T10:25:44.113 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T10:25:44.151 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T10:25:44.496 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T10:25:44.512 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:44.554 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:44.947 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T10:25:44.964 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:45.005 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:45.525 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T10:25:45.545 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:45.583 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:46.428 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T10:25:46.447 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:46.487 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:48.539 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph. 2024-09-18T10:25:48.555 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:48.606 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:48.950 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T10:25:48.968 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:49.008 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:49.436 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T10:25:49.452 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:49.494 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:50.798 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephadm. 2024-09-18T10:25:50.814 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:25:50.855 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:51.150 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T10:25:51.170 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T10:25:51.233 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T10:25:51.552 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T10:25:51.568 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:25:51.610 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:51.963 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T10:25:51.980 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T10:25:52.021 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T10:25:52.282 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T10:25:52.299 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T10:25:52.340 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T10:25:52.609 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T10:25:52.629 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T10:25:52.667 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T10:25:53.012 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-routes. 2024-09-18T10:25:53.028 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T10:25:53.070 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T10:25:53.389 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T10:25:53.405 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:25:53.447 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:54.778 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T10:25:54.793 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T10:25:54.835 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T10:25:55.139 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T10:25:55.156 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T10:25:55.197 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T10:25:55.542 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T10:25:55.559 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T10:25:55.600 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T10:25:56.883 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T10:25:56.900 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T10:25:56.941 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T10:25:57.486 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T10:25:57.504 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T10:25:57.544 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T10:25:57.939 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T10:25:57.955 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T10:25:57.997 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T10:25:58.672 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T10:25:58.691 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:25:58.729 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:25:59.874 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T10:25:59.894 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T10:25:59.932 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T10:26:00.201 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T10:26:00.221 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T10:26:00.251 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T10:26:00.562 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T10:26:00.578 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T10:26:00.621 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T10:26:00.898 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T10:26:00.918 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T10:26:00.965 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T10:26:01.309 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T10:26:01.329 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T10:26:01.369 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T10:26:01.671 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T10:26:01.690 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T10:26:01.729 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T10:26:02.057 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T10:26:02.075 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T10:26:02.123 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T10:26:02.452 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T10:26:02.469 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T10:26:02.510 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T10:26:02.829 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T10:26:02.845 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T10:26:02.888 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T10:26:03.207 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T10:26:03.224 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T10:26:03.265 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T10:26:03.967 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T10:26:03.984 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:26:04.028 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:04.344 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T10:26:04.361 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T10:26:04.394 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T10:26:04.739 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T10:26:04.758 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T10:26:04.797 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:26:05.075 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package jq. 2024-09-18T10:26:05.091 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T10:26:05.133 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:26:05.411 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package socat. 2024-09-18T10:26:05.428 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T10:26:05.469 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T10:26:05.806 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T10:26:05.824 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T10:26:05.864 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T10:26:06.547 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-test. 2024-09-18T10:26:06.564 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:26:06.605 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:11.184 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T10:26:11.200 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T10:26:11.242 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:11.548 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T10:26:11.565 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T10:26:11.606 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:26:12.160 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T10:26:12.177 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:26:12.217 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:12.612 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T10:26:12.629 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T10:26:12.688 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T10:26:13.025 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T10:26:13.041 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T10:26:13.083 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:26:14.700 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T10:26:14.718 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T10:26:14.757 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T10:26:15.102 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T10:26:15.119 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T10:26:15.160 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T10:26:15.446 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-attr. 2024-09-18T10:26:15.467 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T10:26:15.504 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T10:26:15.848 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T10:26:15.868 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T10:26:15.898 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T10:26:16.201 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T10:26:16.220 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T10:26:16.259 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T10:26:16.637 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T10:26:16.657 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T10:26:16.695 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T10:26:17.132 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T10:26:17.151 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T10:26:17.181 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T10:26:17.509 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T10:26:17.528 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T10:26:17.567 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T10:26:17.862 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T10:26:17.879 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T10:26:17.920 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T10:26:18.281 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-py. 2024-09-18T10:26:18.302 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T10:26:18.339 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T10:26:18.642 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T10:26:18.663 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T10:26:18.700 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T10:26:18.994 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T10:26:19.011 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T10:26:19.052 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T10:26:19.389 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T10:26:19.406 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T10:26:19.447 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T10:26:19.784 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T10:26:19.801 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T10:26:19.841 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T10:26:20.144 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T10:26:20.161 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T10:26:20.202 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T10:26:20.917 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package radosgw. 2024-09-18T10:26:20.936 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:26:20.975 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:22.696 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T10:26:22.713 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T10:26:22.753 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:23.056 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T10:26:23.375 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T10:26:23.500 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T10:26:23.626 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T10:26:23.836 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:24.400 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user cephadm....done 2024-09-18T10:26:24.561 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:26:24.696 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T10:26:25.072 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T10:26:25.339 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T10:26:25.449 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T10:26:25.600 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T10:26:25.919 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T10:26:26.381 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T10:26:26.672 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T10:26:26.932 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T10:26:27.217 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T10:26:27.525 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:27.826 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T10:26:27.952 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T10:26:28.270 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T10:26:28.555 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T10:26:28.656 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T10:26:29.007 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T10:26:29.300 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T10:26:29.584 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T10:26:29.935 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T10:26:30.211 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T10:26:30.546 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T10:26:30.830 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T10:26:31.098 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T10:26:31.199 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T10:26:31.358 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T10:26:31.643 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T10:26:32.069 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T10:26:32.446 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T10:26:32.814 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T10:26:32.965 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:33.275 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:26:33.409 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T10:26:33.760 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T10:26:33.886 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T10:26:34.012 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T10:26:34.313 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T10:26:34.598 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:26:34.707 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T10:26:34.825 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T10:26:35.151 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T10:26:35.452 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T10:26:35.754 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T10:26:35.880 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T10:26:36.156 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T10:26:36.432 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T10:26:36.699 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T10:26:37.032 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T10:26:37.552 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T10:26:37.678 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T10:26:38.029 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T10:26:38.397 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T10:26:38.498 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T10:26:38.624 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T10:26:38.942 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T10:26:39.235 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:39.361 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T10:26:40.849 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:40.975 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:26:41.135 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T10:26:41.503 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T10:26:41.804 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T10:26:42.113 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T10:26:42.223 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T10:26:42.541 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:42.666 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T10:26:42.993 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:43.119 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T10:26:43.488 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T10:26:44.517 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:44.618 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T10:26:44.728 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T10:26:44.854 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T10:26:44.980 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T10:26:45.281 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T10:26:45.407 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T10:26:45.708 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:45.817 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:46.046 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T10:26:46.046 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T10:26:46.485 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:46.680 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T10:26:46.989 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T10:26:47.316 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T10:26:47.668 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:47.794 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:47.929 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:48.046 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:48.155 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T10:26:48.448 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T10:26:48.750 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:48.875 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:49.001 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T10:26:49.905 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:50.473 INFO:teuthology.orchestra.run.smithi174.stdout:Adding group ceph....done 2024-09-18T10:26:50.842 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user ceph....done 2024-09-18T10:26:51.026 INFO:teuthology.orchestra.run.smithi174.stdout:Setting system user ceph properties....done 2024-09-18T10:26:51.054 INFO:teuthology.orchestra.run.smithi174.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T10:26:51.201 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T10:26:51.506 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T10:26:51.931 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:52.057 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:52.331 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T10:26:52.331 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T10:26:52.967 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:53.345 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T10:26:53.802 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:54.029 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T10:26:54.029 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T10:26:54.480 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:54.713 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T10:26:54.714 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T10:26:55.156 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:55.650 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T10:26:55.650 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T10:26:56.100 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:56.226 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:56.368 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:56.595 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T10:26:56.596 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T10:26:57.044 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:57.187 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:57.313 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:57.489 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T10:26:57.616 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T10:26:57.767 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T10:26:58.145 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T10:27:00.972 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T10:27:01.572 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T10:27:01.577 DEBUG:teuthology.parallel:result is None 2024-09-18T10:27:01.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:27:01.735 DEBUG:teuthology.orchestra.run.smithi174:> dpkg-query -W -f '${Version}' ceph 2024-09-18T10:27:01.759 INFO:teuthology.orchestra.run.smithi174.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:27:01.759 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T10:27:01.759 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T10:27:01.761 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T10:27:01.761 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:27:01.761 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T10:27:01.816 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T10:27:01.816 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:27:01.816 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T10:27:01.827 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T10:27:01.838 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T10:27:01.839 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:27:01.839 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T10:27:01.852 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T10:27:01.864 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T10:27:01.864 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:27:01.864 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T10:27:01.921 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T10:27:01.930 INFO:teuthology.run_tasks:Running task workunit... 2024-09-18T10:27:01.947 INFO:tasks.workunit:Pulling workunits from ref 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:27:01.948 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-18T10:27:01.948 INFO:tasks.workunit:timeout=3h 2024-09-18T10:27:01.948 INFO:tasks.workunit:cleanup=True 2024-09-18T10:27:01.949 DEBUG:teuthology.orchestra.run.smithi174:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-18T10:27:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T10:27:01.955 INFO:teuthology.orchestra.run.smithi174.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-18T10:27:01.955 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T10:27:02.003 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-18T10:27:02.003 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-18T10:27:02.010 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T10:27:02.018 INFO:tasks.workunit.client.0.smithi174.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-18T10:27:54.413 INFO:tasks.workunit.client.0.smithi174.stderr:Note: switching to '7061ea6ca50908d104e9f3979dc615e118d9e153'. 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr:state without impacting any branches by switching back to a branch. 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: git switch -c 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr:Or undo this operation with: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.414 INFO:tasks.workunit.client.0.smithi174.stderr: git switch - 2024-09-18T10:27:54.415 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.415 INFO:tasks.workunit.client.0.smithi174.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-18T10:27:54.415 INFO:tasks.workunit.client.0.smithi174.stderr: 2024-09-18T10:27:54.415 INFO:tasks.workunit.client.0.smithi174.stderr:HEAD is now at 7061ea6ca50 Merge branch 'wip-65946-quincy' of https://github.com/amathuria/ceph into wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T10:27:54.419 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-18T10:27:54.426 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-18T10:27:54.427 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-18T10:27:54.431 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-09-18T10:27:54.431 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-09-18T10:27:54.432 DEBUG:teuthology.orchestra.run.smithi174:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-09-18T10:27:54.487 INFO:tasks.workunit.client.0.smithi174.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-18T10:27:54.487 INFO:tasks.workunit.client.0.smithi174.stderr:++ TIMEOUT=300 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ MAX_TIMEOUT=15 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ PG_NUM=4 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ TMPDIR=/tmp 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ type xmlstarlet 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:++ XMLSTARLET=xmlstarlet 2024-09-18T10:27:54.488 INFO:tasks.workunit.client.0.smithi174.stderr:+++ uname 2024-09-18T10:27:54.489 INFO:tasks.workunit.client.0.smithi174.stderr:++ '[' Linux = FreeBSD ']' 2024-09-18T10:27:54.489 INFO:tasks.workunit.client.0.smithi174.stderr:++ SED=sed 2024-09-18T10:27:54.489 INFO:tasks.workunit.client.0.smithi174.stderr:++ AWK=awk 2024-09-18T10:27:54.490 INFO:tasks.workunit.client.0.smithi174.stderr:+++ stty -a 2024-09-18T10:27:54.490 INFO:tasks.workunit.client.0.smithi174.stderr:+++ head -1 2024-09-18T10:27:54.490 INFO:tasks.workunit.client.0.smithi174.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-18T10:27:54.511 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T10:27:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:++ termwidth= 2024-09-18T10:27:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-18T10:27:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:++ DIFFCOLOPTS='-y ' 2024-09-18T10:27:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:++ KERNCORE=kernel.core_pattern 2024-09-18T10:27:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:++ EXTRA_OPTS= 2024-09-18T10:27:54.523 INFO:tasks.workunit.client.0.smithi174.stderr:++ test '' = TESTS 2024-09-18T10:27:54.524 INFO:tasks.workunit.client.0.smithi174.stderr:++ uname -m 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ arch=x86_64 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ case $arch in 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ plugins=(jerasure shec lrc isa) 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ main test-erasure-code-plugins 2024-09-18T10:27:54.527 INFO:tasks.workunit.client.0.smithi174.stderr:+ local dir=td/test-erasure-code-plugins 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:+ shift 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:+ shopt -s -o xtrace 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T10:27:54.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-erasure-code-plugins 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-09-18T10:27:54.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none ' 2024-09-18T10:27:54.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-09-18T10:27:54.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-09-18T10:27:54.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_preload_no_warning 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_preload_no_warning_default 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_preload_warning' 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:54.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-09-18T10:27:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:27:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:27:54.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:27:54.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:27:54.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:27:54.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:27:54.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:27:54.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:27:54.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:27:54.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:27:54.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:27:54.538 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:27:54.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:27:54.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:27:54.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:27:54.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:27:54.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:27:54.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:27:54.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:27:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:27:54.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:27:54.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:27:54.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:27:54.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:54.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:54.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:27:54.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:27:54.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:27:54.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:27:54.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:27:54.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:54.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:54.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:54.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:27:54.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:27:54.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:27:54.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:27:54.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-18T10:27:55.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:27:55.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:27:55.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:27:55.049 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:27:55.049 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:55.049 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:55.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:27:55.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T10:27:55.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:27:55.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:27:55.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:27:55.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:27:55.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:27:55.299 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:27:55.299 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:27:55.299 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:27:55.300 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:27:55.300 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:27:55.300 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:55.300 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:55.301 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:27:55.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:27:55.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:27:55.390 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:27:55.390 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:27:55.390 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:27:55.390 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:27:55.391 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:27:55.391 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:27:55.391 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:27:55.391 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:27:55.391 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:27:55.391 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:55.392 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:55.392 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:27:55.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:27:55.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:27:55.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:27:56.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:27:56.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:27:56.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:27:56.361 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:27:56.361 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:56.361 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:56.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:27:56.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:27:56.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:27:56.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-09-18T10:27:56.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-18T10:27:56.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-09-18T10:27:56.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:27:56.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:27:56.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:27:56.457 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:27:56.457 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:27:56.457 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:27:56.458 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:27:56.458 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:27:56.458 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:27:56.458 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:27:56.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:27:56.460 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:27:56.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75a5c21b-cd99-4490-893a-0628979ffefc 2024-09-18T10:27:56.461 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 75a5c21b-cd99-4490-893a-0628979ffefc 2024-09-18T10:27:56.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 75a5c21b-cd99-4490-893a-0628979ffefc' 2024-09-18T10:27:56.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:27:56.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsq+pmuHOcHBAA38Kfe2RW/O0zWdM9NxrrJg== 2024-09-18T10:27:56.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsq+pmuHOcHBAA38Kfe2RW/O0zWdM9NxrrJg=="}' 2024-09-18T10:27:56.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75a5c21b-cd99-4490-893a-0628979ffefc -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:27:56.850 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:27:56.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:27:56.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsq+pmuHOcHBAA38Kfe2RW/O0zWdM9NxrrJg== --osd-uuid 75a5c21b-cd99-4490-893a-0628979ffefc 2024-09-18T10:27:56.912 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:56.907+0000 7fe8f3449540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:27:56.933 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:56.927+0000 7fe8f3449540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:27:56.935 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:56.931+0000 7fe8f3449540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:27:56.935 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:56.931+0000 7fe8f3449540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:27:56.936 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:56.931+0000 7fe8f3449540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:27:56.936 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:56.931+0000 7fe8f3449540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:27:57.034 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:27:57.031+0000 7fe8f3449540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:28:02.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:28:02.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:28:02.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:28:02.972 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:28:02.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:28:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:28:03.200 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:28:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:28:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:28:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:28:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:28:03.229 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:03.223+0000 7fbd73fbf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:03.231 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:03.227+0000 7fbd73fbf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:03.233 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:03.227+0000 7fbd73fbf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:03.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:28:03.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:28:03.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:28:03.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:28:03.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:28:03.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:03.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:28:03.413 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:28:03.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:03.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:03.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:03.799 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:03.795+0000 7fbd73fbf540 -1 Falling back to public interface 2024-09-18T10:28:04.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:04.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:04.639 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:28:04.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:28:04.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:04.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:05.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:06.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:06.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:06.023 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:28:06.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:28:06.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:06.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:06.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:06.632 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:06.627+0000 7fbd73fbf540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:28:07.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:07.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:07.466 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:28:07.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:28:07.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:07.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:07.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:08.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:08.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:08.976 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:28:08.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:28:08.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:08.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:09.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:10.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:10.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:10.491 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:28:10.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:28:10.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:10.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:10.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:11.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:11.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:11.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:28:11.901 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:28:11.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:11.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:12.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:13.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:13.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:13.424 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:28:13.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:28:13.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:13.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:13.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:14.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:14.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:14.851 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:28:14.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:28:14.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:14.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:15.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:16.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:16.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:16.413 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:28:16.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:28:16.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:16.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:16.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:17.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:17.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:17.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:28:17.830 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:28:17.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:17.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:18.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:19.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:19.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:19.294 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:28:19.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:28:19.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:19.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:20.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:21.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:21.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:21.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:28:21.301 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:28:21.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:21.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:22.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:23.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:23.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:23.430 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:28:23.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:28:23.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:23.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:24.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:25.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:25.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:25.287 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:28:25.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:28:25.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:25.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:25.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:26.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:26.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:26.719 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:28:26.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:28:26.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:26.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:27.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:28.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:28.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:28.221 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:28:28.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:28:28.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:28.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:28.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:29.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:29.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:29.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:28:29.627 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:28:29.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:29.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:30.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:31.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:31.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:31.181 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:28:31.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:28:31.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:31.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:28:31.600 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/18233,v1:127.0.0.1:6801/18233] [v2:127.0.0.1:6802/18233,v1:127.0.0.1:6803/18233] exists,up 75a5c21b-cd99-4490-893a-0628979ffefc 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:28:31.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:28:31.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:28:31.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:28:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0072a2e-8be7-4d9f-a667-a89ec467c814 2024-09-18T10:28:31.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c0072a2e-8be7-4d9f-a667-a89ec467c814' 2024-09-18T10:28:31.608 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 c0072a2e-8be7-4d9f-a667-a89ec467c814 2024-09-18T10:28:31.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:28:31.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPq+pmVX7QJBAAH9F1zicY0rvYgxTkALRuig== 2024-09-18T10:28:31.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPq+pmVX7QJBAAH9F1zicY0rvYgxTkALRuig=="}' 2024-09-18T10:28:31.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0072a2e-8be7-4d9f-a667-a89ec467c814 -i td/test-erasure-code-plugins/1/new.json 2024-09-18T10:28:32.222 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:28:32.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-09-18T10:28:32.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPq+pmVX7QJBAAH9F1zicY0rvYgxTkALRuig== --osd-uuid c0072a2e-8be7-4d9f-a667-a89ec467c814 2024-09-18T10:28:32.267 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.262+0000 7fe813f0b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:32.269 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.266+0000 7fe813f0b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:32.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.266+0000 7fe813f0b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:32.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.266+0000 7fe813f0b540 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-18T10:28:32.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.266+0000 7fe813f0b540 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-18T10:28:32.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.266+0000 7fe813f0b540 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-09-18T10:28:32.352 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:32.350+0000 7fe813f0b540 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-09-18T10:28:38.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-09-18T10:28:38.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:28:38.310 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T10:28:38.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T10:28:38.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:28:38.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T10:28:38.845 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T10:28:38.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:28:38.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:28:38.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:28:38.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:28:38.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:38.858+0000 7f6c6b841540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:38.868 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:38.862+0000 7f6c6b841540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:38.869 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:38.862+0000 7f6c6b841540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:28:39.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T10:28:39.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:28:39.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:39.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:39.408 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:39.402+0000 7f6c6b841540 -1 Falling back to public interface 2024-09-18T10:28:39.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:40.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:40.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:40.702 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:28:40.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:28:40.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:40.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:41.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:42.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:42.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:42.102 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:28:42.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:28:42.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:42.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:42.152 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:28:42.146+0000 7f6c6b841540 -1 osd.1 0 log_to_monitors true 2024-09-18T10:28:42.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:43.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:43.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:43.596 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:28:43.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:28:43.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:43.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:44.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:45.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:45.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:45.058 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:28:45.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:28:45.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:45.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:45.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:46.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:46.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:46.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:28:46.486 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:28:46.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:46.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:46.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:47.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:47.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:47.981 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:28:47.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:28:47.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:47.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:48.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:49.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:49.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:49.382 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:28:49.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:28:49.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:49.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:50.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:50.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:50.877 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:28:50.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:28:50.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:50.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:51.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:52.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:52.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:52.417 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:28:52.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:28:52.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:52.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:52.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:53.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:53.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:53.896 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:28:53.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:28:53.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:53.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:54.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:55.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:55.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:55.845 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:28:55.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:28:55.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:55.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:56.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:57.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:57.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:57.587 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:28:57.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:28:57.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:57.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:28:58.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:28:59.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:28:59.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:28:59.583 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:28:59.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:28:59.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:28:59.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:29:00.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:01.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:01.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:01.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:29:01.345 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:29:01.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:01.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:29:02.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:03.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:03.084 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:29:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:29:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:29:03.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:04.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:04.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:04.759 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:29:04.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:29:04.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:04.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:29:05.191 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/20162,v1:127.0.0.1:6811/20162] [v2:127.0.0.1:6812/20162,v1:127.0.0.1:6813/20162] exists,up c0072a2e-8be7-4d9f-a667-a89ec467c814 2024-09-18T10:29:05.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:29:05.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:29:05.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:29:05.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-09-18T10:29:05.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:29:05.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3299b03a-c18e-424a-841d-8579318e5116 2024-09-18T10:29:05.200 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 3299b03a-c18e-424a-841d-8579318e5116 2024-09-18T10:29:05.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3299b03a-c18e-424a-841d-8579318e5116' 2024-09-18T10:29:05.200 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:29:05.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxq+pm8cvRDBAAKddZBfXMGgpsdqjSMwpKWQ== 2024-09-18T10:29:05.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxq+pm8cvRDBAAKddZBfXMGgpsdqjSMwpKWQ=="}' 2024-09-18T10:29:05.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3299b03a-c18e-424a-841d-8579318e5116 -i td/test-erasure-code-plugins/2/new.json 2024-09-18T10:29:05.814 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:29:05.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-09-18T10:29:05.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxq+pm8cvRDBAAKddZBfXMGgpsdqjSMwpKWQ== --osd-uuid 3299b03a-c18e-424a-841d-8579318e5116 2024-09-18T10:29:05.860 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.853+0000 7fb88b3ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:05.863 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.857+0000 7fb88b3ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:05.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.857+0000 7fb88b3ae540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:05.866 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.861+0000 7fb88b3ae540 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-18T10:29:05.866 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.861+0000 7fb88b3ae540 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-18T10:29:05.866 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.861+0000 7fb88b3ae540 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-09-18T10:29:05.943 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:05.937+0000 7fb88b3ae540 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-09-18T10:29:11.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-09-18T10:29:11.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:29:11.962 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T10:29:11.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T10:29:11.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:29:12.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T10:29:12.503 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:29:12.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:29:12.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:29:12.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:29:12.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:29:12.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:12.525+0000 7fee95ea2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:12.567 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:12.561+0000 7fee95ea2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:12.568 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:12.561+0000 7fee95ea2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:12.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T10:29:12.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:29:12.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:29:12.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:29:12.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:29:12.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:12.932 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:29:12.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:29:12.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:12.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:13.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:13.125+0000 7fee95ea2540 -1 Falling back to public interface 2024-09-18T10:29:13.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:14.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:14.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:14.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:29:14.404 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:29:14.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:14.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:14.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:15.836 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:15.829+0000 7fee95ea2540 -1 osd.2 0 log_to_monitors true 2024-09-18T10:29:15.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:15.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:15.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:29:15.857 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:29:15.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:15.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:16.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:17.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:17.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:17.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:29:17.651 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:29:17.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:17.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:18.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:19.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:19.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:19.104 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:29:19.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:29:19.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:19.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:19.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:20.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:20.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:20.506 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:29:20.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:29:20.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:20.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:21.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:22.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:22.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:22.128 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:29:22.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:29:22.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:22.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:22.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:23.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:23.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:23.552 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:29:23.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:29:23.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:23.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:24.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:25.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:25.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:25.028 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:29:25.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:29:25.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:25.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:25.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:26.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:26.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:26.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:29:26.428 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:29:26.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:26.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:27.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:28.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:29:28.009 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:29:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:28.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:29.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:29.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:29.412 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:29:29.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:29:29.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:29.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:30.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:31.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:31.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:31.154 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:29:31.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:29:31.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:31.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:31.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:32.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:32.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:32.812 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:29:32.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:29:32.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:32.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:33.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:34.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:34.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:34.443 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:29:34.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:29:34.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:34.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:35.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:36.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:36.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:36.259 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:29:36.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:29:36.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:36.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:37.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:38.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:38.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:38.134 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:29:38.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:29:38.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:38.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:38.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:29:39.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:29:39.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:29:39.590 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:29:39.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:29:39.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:29:39.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:29:40.050 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/21928,v1:127.0.0.1:6819/21928] [v2:127.0.0.1:6820/21928,v1:127.0.0.1:6821/21928] exists,up 3299b03a-c18e-424a-841d-8579318e5116 2024-09-18T10:29:40.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:29:40.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:29:40.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:29:40.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-09-18T10:29:40.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T10:29:40.459 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' does not exist 2024-09-18T10:29:40.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T10:29:40.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T10:29:41.168 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T10:29:41.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:29:42.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T10:29:43.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-09-18T10:29:43.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:29:43.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:29:43.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:29:43.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:29:43.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:29:43.542 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:29:43.542 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:29:43.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:29:43.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:29:43.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:29:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:29:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:29:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:29:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:29:43.743 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:29:44.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:29:44.198 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:29:44.198 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T10:29:44.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:29:44.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:29:44.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:29:44.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-18T10:29:44.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-18T10:29:44.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-18T10:29:44.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:29:44.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:29:44.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-18T10:29:44.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-18T10:29:44.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-09-18T10:29:44.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:29:44.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:29:44.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-18T10:29:44.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-18T10:29:44.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509453' 2024-09-18T10:29:44.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:29:44.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-18T10:29:44.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:29:44.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:29:44.652 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-18T10:29:44.653 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:29:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-18T10:29:44.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-18T10:29:44.654 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836500 2024-09-18T10:29:44.654 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:29:45.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-18T10:29:45.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T10:29:46.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T10:29:46.086 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:29:46.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-18T10:29:46.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:29:46.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:29:46.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-18T10:29:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:29:46.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-18T10:29:46.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:29:46.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-18T10:29:46.527 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949672976 2024-09-18T10:29:46.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-18T10:29:46.527 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:29:47.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-18T10:29:47.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:29:47.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-18T10:29:47.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:29:47.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:29:47.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-18T10:29:47.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:29:47.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-18T10:29:47.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-18T10:29:47.085 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509453 2024-09-18T10:29:47.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:29:47.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-18T10:29:47.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:29:47.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:29:47.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:29:48.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T10:29:48.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:29:48.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:29:48.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:29:48.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:29:48.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:29:48.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:29:48.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:29:48.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T10:29:48.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:29:48.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:29:48.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:29:48.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T10:29:48.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:29:48.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:29:48.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-18T10:29:48.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-09-18T10:29:49.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-18T10:29:49.562 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:49.562 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:49.563 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:49.563 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:49.563 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:49.563 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:49.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-18T10:29:49.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:49.632 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:49.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:29:49.651 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:49.228+0000 7fb4b28cc700 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:49.651 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:49.540+0000 7fb4b28cc700 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:49.651 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:49.540+0000 7fb4b28cc700 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:49.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-18T10:29:49.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-09-18T10:29:50.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-18T10:29:50.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:50.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:50.263 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:50.263 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:50.263 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:50.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:50.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-18T10:29:50.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:50.328 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:50.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:29:50.348 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:49.928+0000 7fb4b28cc700 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:50.348 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:50.244+0000 7fb4b28cc700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:50.349 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:50.244+0000 7fb4b28cc700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:50.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-09-18T10:29:50.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-09-18T10:29:50.960 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-18T10:29:50.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:50.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:50.961 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:50.961 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:50.961 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:50.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:50.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-18T10:29:50.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:51.036 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:51.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:29:51.056 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:50.632+0000 7fb4b28cc700 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:51.056 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:50.940+0000 7fb4b28cc700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:51.056 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:50.940+0000 7fb4b28cc700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-09-18T10:29:51.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-18T10:29:51.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-09-18T10:29:51.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-18T10:29:51.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:51.664 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:51.664 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:51.664 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:51.664 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:51.665 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:51.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-18T10:29:51.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:51.745 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:51.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:29:51.763 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:51.364+0000 7fb4b28cc700 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-18T10:29:51.764 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:51.640+0000 7fb4b28cc700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-18T10:29:51.764 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:51.640+0000 7fb4b28cc700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-09-18T10:29:51.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-18T10:29:51.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-09-18T10:29:52.345 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-18T10:29:52.345 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:52.345 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:52.346 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:52.346 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:52.346 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:52.346 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:52.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-18T10:29:52.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:52.419 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:52.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:29:52.437 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:52.052+0000 7fb4b28cc700 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-18T10:29:52.437 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:52.324+0000 7fb4b28cc700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-18T10:29:52.437 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:52.324+0000 7fb4b28cc700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-09-18T10:29:52.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-09-18T10:29:52.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-09-18T10:29:53.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-09-18T10:29:53.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:53.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:53.042 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:53.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-09-18T10:29:53.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:53.114 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:53.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:29:53.135 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:52.736+0000 7fb4b28cc700 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-18T10:29:53.135 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:53.016+0000 7fb4b28cc700 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-18T10:29:53.135 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:29:53.016+0000 7fb4b28cc700 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-09-18T10:29:53.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:29:53.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:53.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:29:53.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:29:53.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:29:53.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:29:53.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:29:53.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:29:53.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:29:53.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:29:53.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:29:53.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:29:53.256 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:29:53.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:29:53.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:29:53.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:29:53.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:29:53.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:29:53.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:29:53.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:29:53.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:29:53.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:29:53.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:29:53.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:29:53.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:53.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:53.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:29:53.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:29:53.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:29:53.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-18T10:29:53.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-09-18T10:29:53.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:29:53.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:29:53.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:29:53.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:29:53.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:29:53.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:29:53.290 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:29:53.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:29:53.292 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:29:53.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:29:53.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:29:53.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:29:53.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:29:53.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:29:53.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:29:53.296 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:29:53.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:29:53.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:29:53.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:29:53.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:29:53.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:53.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:53.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:29:53.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:29:53.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:29:53.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:29:53.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:29:53.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:53.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:53.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:29:53.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:29:53.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:29:53.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-09-18T10:29:53.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:29:53.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:29:53.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:29:53.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:29:53.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:53.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:53.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:29:53.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-09-18T10:29:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:29:53.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:29:53.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:29:53.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:29:53.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:29:53.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:29:53.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:29:53.964 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:53.964 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:53.965 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:53.965 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:53.965 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:53.965 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:53.966 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:29:53.966 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:29:54.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:29:54.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:29:54.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:29:54.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:29:54.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:29:54.058 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:29:54.058 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:54.058 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:54.059 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:54.059 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:54.059 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:54.059 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:54.059 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:29:54.060 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:29:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:29:55.038 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:29:55.038 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:29:55.038 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:29:55.039 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:29:55.039 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:55.039 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:55.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:29:55.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:29:55.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:29:55.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-18T10:29:55.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:29:55.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:29:55.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:29:55.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:55.074 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:55.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:29:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:29:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:29:55.153 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:29:55.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:29:55.163 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:29:55.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:29:55.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-09-18T10:29:55.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:29:55.166 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:29:55.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df41ba13-0508-4430-924a-8e47bdf3172a 2024-09-18T10:29:55.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 df41ba13-0508-4430-924a-8e47bdf3172a' 2024-09-18T10:29:55.166 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 df41ba13-0508-4430-924a-8e47bdf3172a 2024-09-18T10:29:55.166 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:29:55.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjq+pmPTDiChAAUzIVUVqbMV3ul07qzJIzSQ== 2024-09-18T10:29:55.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjq+pmPTDiChAAUzIVUVqbMV3ul07qzJIzSQ=="}' 2024-09-18T10:29:55.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df41ba13-0508-4430-924a-8e47bdf3172a -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:29:55.539 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:29:55.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:29:55.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQCjq+pmPTDiChAAUzIVUVqbMV3ul07qzJIzSQ== --osd-uuid df41ba13-0508-4430-924a-8e47bdf3172a 2024-09-18T10:29:55.584 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.580+0000 7f98ee502540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:55.587 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.584+0000 7f98ee502540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:55.590 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.584+0000 7f98ee502540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:29:55.590 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.584+0000 7f98ee502540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:29:55.590 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.584+0000 7f98ee502540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:29:55.590 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.584+0000 7f98ee502540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:29:55.665 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:29:55.660+0000 7f98ee502540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:30:01.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:30:01.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:30:01.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:30:01.961 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:30:01.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:30:02.547 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:30:02.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:30:02.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-09-18T10:30:02.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:30:02.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:30:02.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:30:02.575 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:02.572+0000 7f2b65746540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:02.578 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:02.572+0000 7f2b65746540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:02.579 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:02.576+0000 7f2b65746540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:02.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:30:02.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:30:02.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:30:02.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:30:02.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:30:02.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:02.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:30:02.986 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:30:02.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:02.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:03.116 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:03.112+0000 7f2b65746540 -1 Falling back to public interface 2024-09-18T10:30:03.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:04.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:04.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:04.422 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:30:04.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:30:04.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:04.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:04.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:05.643 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:05.640+0000 7f2b65746540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:30:05.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:05.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:05.819 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:30:05.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:30:05.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:05.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:06.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:07.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:07.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:07.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:30:07.492 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:30:07.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:07.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:07.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:08.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:08.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:08.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:30:08.903 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:30:08.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:08.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:09.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:10.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:10.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:10.418 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:30:10.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:30:10.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:10.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:10.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:11.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:11.821 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:30:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:30:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:11.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:12.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:13.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:13.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:13.302 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:30:13.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:30:13.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:13.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:13.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:14.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:14.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:14.750 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:30:14.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:30:14.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:14.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:15.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:16.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:16.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:16.149 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:30:16.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:30:16.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:16.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:16.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:17.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:17.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:17.672 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:30:17.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:30:17.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:17.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:18.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:19.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:19.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:19.141 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:30:19.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:30:19.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:19.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:19.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:20.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:20.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:20.915 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:30:20.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:30:20.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:20.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:21.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:22.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:22.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:22.713 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:30:22.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:30:22.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:22.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:23.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:24.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:24.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:24.434 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:30:24.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:30:24.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:24.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:25.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:26.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:26.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:26.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:30:26.273 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:30:26.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:26.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:27.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:27.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:27.956 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:30:27.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:30:27.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:27.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:28.394 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/24647,v1:127.0.0.1:6803/24647] [v2:127.0.0.1:6804/24647,v1:127.0.0.1:6805/24647] exists,up df41ba13-0508-4430-924a-8e47bdf3172a 2024-09-18T10:30:28.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:30:28.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:30:28.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:30:28.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-18T10:30:28.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:30:28.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:30:28.396 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:30:28.397 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:28.397 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:28.397 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:30:28.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:30:28.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:30:28.473 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:30:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:30:28.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:30:28.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:30:28.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:30:28.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:30:28.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:30:28.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:30:28.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:30:28.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:30:28.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:30:28.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:30:28.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:30:28.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:30:28.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:30:28.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:30:28.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:30:28.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:30:28.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:30:28.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:30:28.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:30:28.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:30:28.611 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:30:28.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:30:28.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:30:28.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:30:28.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:30:28.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:28.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:28.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:30:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:30:28.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:30:28.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:30:28.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:30:28.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:30:28.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:30:28.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:30:28.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:30:28.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:30:28.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:30:28.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:30:28.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:30:28.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:30:28.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:30:28.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:30:28.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:30:28.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:30:28.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:28.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:28.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:30:28.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:30:28.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:30:28.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:30:28.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:30:28.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:28.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:28.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:30:28.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:30:28.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:30:28.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:30:28.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:30:28.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-09-18T10:30:28.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:30:28.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:30:28.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:30:28.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:30:28.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:30:28.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-09-18T10:30:29.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:30:29.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:30:29.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:30:29.065 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:30:29.065 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:29.065 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:29.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:30:29.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-09-18T10:30:29.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:30:29.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:30:29.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:30:29.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:30:29.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:30:29.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:30:29.338 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:30:29.338 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:30:29.338 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:30:29.338 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:30:29.339 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:29.339 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:29.339 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:30:29.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:30:29.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:30:29.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:30:29.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:30:29.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:30:29.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:30:29.432 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:30:29.432 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:30:29.432 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:30:29.432 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:30:29.434 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:30:29.434 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:29.434 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:29.434 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:30:29.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:30:29.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:30:29.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:30:29.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:30:29.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:30:29.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:30:29.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:30:29.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:30:29.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:30:30.409 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:30:30.409 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:30:30.409 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:30:30.410 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:30:30.410 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:30.410 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:30.410 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:30:30.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:30:30.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:30:30.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-18T10:30:30.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:30:30.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:30:30.445 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:30:30.446 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:30.446 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:30.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:30:30.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:30:30.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:30:30.526 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:30:30.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:30:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:30:30.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:30:30.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:30:30.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:30:30.538 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:30:30.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:30:30.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:30:30.540 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:30:30.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:30:30.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:30:30.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-09-18T10:30:30.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:30:30.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:30:30.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ed17847e-b6c2-445a-893d-3629979ed8e3 2024-09-18T10:30:30.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ed17847e-b6c2-445a-893d-3629979ed8e3' 2024-09-18T10:30:30.543 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 ed17847e-b6c2-445a-893d-3629979ed8e3 2024-09-18T10:30:30.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:30:30.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGq+pmyuEbIRAAymn902KAbRJciNmOUON1zw== 2024-09-18T10:30:30.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGq+pmyuEbIRAAymn902KAbRJciNmOUON1zw=="}' 2024-09-18T10:30:30.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ed17847e-b6c2-445a-893d-3629979ed8e3 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:30:30.911 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:30:30.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:30:30.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQDGq+pmyuEbIRAAymn902KAbRJciNmOUON1zw== --osd-uuid ed17847e-b6c2-445a-893d-3629979ed8e3 2024-09-18T10:30:30.954 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:30.947+0000 7f06d99b1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:30.957 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:30.951+0000 7f06d99b1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:30.959 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:30.955+0000 7f06d99b1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:30.960 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:30.955+0000 7f06d99b1540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:30:30.960 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:30.955+0000 7f06d99b1540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:30:30.960 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:30.955+0000 7f06d99b1540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:30:31.034 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:31.027+0000 7f06d99b1540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:30:36.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:30:36.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:30:36.874 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:30:36.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:30:36.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:30:37.416 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:30:37.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:30:37.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-09-18T10:30:37.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:30:37.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:30:37.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:30:37.446 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:37.439+0000 7f9a3884f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:37.465 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:37.459+0000 7f9a3884f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:37.466 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:37.459+0000 7f9a3884f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:37.858 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:30:37.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:30:37.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:37.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:38.021 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:38.015+0000 7f9a3884f540 -1 Falling back to public interface 2024-09-18T10:30:38.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:39.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:39.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:39.243 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:30:39.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:30:39.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:39.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:39.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:40.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:40.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:40.630 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:30:40.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:30:40.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:40.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:30:40.719+0000 7f9a3884f540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:30:41.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:42.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:42.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:42.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:30:42.090 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:30:42.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:42.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:42.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:43.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:43.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:43.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:30:43.560 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:30:43.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:43.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:43.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:30:44.950 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:30:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:44.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:45.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:46.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:46.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:46.505 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:30:46.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:30:46.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:46.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:46.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:47.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:47.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:47.915 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:30:47.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:30:47.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:47.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:48.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:49.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:49.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:49.408 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:30:49.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:30:49.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:49.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:49.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:50.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:50.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:50.845 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:30:50.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:30:50.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:50.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:51.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:52.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:52.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:52.418 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:30:52.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:30:52.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:52.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:53.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:54.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:54.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:54.024 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:30:54.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:30:54.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:54.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:54.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:55.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:55.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:55.776 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:30:55.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:30:55.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:55.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:56.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:57.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:57.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:57.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:30:57.472 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:30:57.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:57.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:30:58.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:30:59.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:30:59.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:30:59.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:30:59.500 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:30:59.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:30:59.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:00.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:01.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:01.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:01.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:31:01.441 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:31:01.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:01.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:02.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:03.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:03.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:03.088 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:31:03.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:31:03.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:03.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:03.495 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/26701,v1:127.0.0.1:6803/26701] [v2:127.0.0.1:6804/26701,v1:127.0.0.1:6805/26701] exists,up ed17847e-b6c2-445a-893d-3629979ed8e3 2024-09-18T10:31:03.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:31:03.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:31:03.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:31:03.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-18T10:31:03.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:31:03.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:31:03.498 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:03.498 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:03.498 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:03.498 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:31:03.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:31:03.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:31:03.575 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:31:03.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:31:03.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:31:03.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:31:03.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:03.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:31:03.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:31:03.590 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:31:03.590 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:31:03.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:31:03.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:31:03.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:31:03.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:31:03.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:31:03.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:31:03.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:31:03.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:31:03.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:31:03.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:31:03.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:03.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:31:03.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:31:03.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:03.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:31:03.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:31:03.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:31:03.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:31:03.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:03.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:03.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:31:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:31:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:31:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:31:03.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:31:03.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:31:03.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:31:03.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:31:03.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:31:03.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:31:03.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:31:03.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:03.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:31:03.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:31:03.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:03.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:31:03.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:31:03.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:31:03.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:31:03.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:03.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:03.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:31:03.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:31:03.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:31:03.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:31:03.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:31:03.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:03.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:03.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:31:03.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:31:03.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:31:03.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:31:03.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-09-18T10:31:04.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:31:04.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:31:04.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:31:04.108 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:31:04.108 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:04.108 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:04.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:04.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-09-18T10:31:04.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:31:04.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:31:04.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:31:04.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:31:04.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:31:04.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:31:04.416 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:31:04.416 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:31:04.416 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:31:04.417 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:04.417 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:04.417 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:04.417 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:31:04.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:31:04.418 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:31:04.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:31:04.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:31:04.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:31:04.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:31:04.513 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:31:04.513 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:31:04.513 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:31:04.513 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:31:04.514 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:04.514 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:04.514 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:04.514 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:31:04.515 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:31:04.515 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:31:04.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:31:05.479 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:31:05.479 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:31:05.479 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:31:05.479 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:31:05.479 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:05.479 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:05.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:05.481 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:31:05.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:31:05.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-18T10:31:05.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:31:05.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:31:05.514 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:05.515 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:05.515 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:05.515 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:31:05.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:31:05.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:31:05.595 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:31:05.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-09-18T10:31:05.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:05.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:31:05.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:31:05.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:31:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:31:05.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:31:05.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:31:05.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:31:05.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-09-18T10:31:05.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:31:05.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:31:05.608 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 96e56ad6-4ed7-4ee4-aa35-49e67cad47b5 2024-09-18T10:31:05.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=96e56ad6-4ed7-4ee4-aa35-49e67cad47b5 2024-09-18T10:31:05.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 96e56ad6-4ed7-4ee4-aa35-49e67cad47b5' 2024-09-18T10:31:05.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:31:05.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpq+pmQd9QJRAAVGOKe1FH2STh0acp4XEaSg== 2024-09-18T10:31:05.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpq+pmQd9QJRAAVGOKe1FH2STh0acp4XEaSg=="}' 2024-09-18T10:31:05.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 96e56ad6-4ed7-4ee4-aa35-49e67cad47b5 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:31:05.974 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:31:05.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:31:05.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQDpq+pmQd9QJRAAVGOKe1FH2STh0acp4XEaSg== --osd-uuid 96e56ad6-4ed7-4ee4-aa35-49e67cad47b5 2024-09-18T10:31:06.022 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.015+0000 7f75b39df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:06.025 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.019+0000 7f75b39df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:06.027 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.023+0000 7f75b39df540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.023+0000 7f75b39df540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:31:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.023+0000 7f75b39df540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:31:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.023+0000 7f75b39df540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:31:06.106 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:06.099+0000 7f75b39df540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:31:12.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:31:12.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:31:12.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:31:12.394 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:31:12.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:31:12.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:31:12.907 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:31:12.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:31:12.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-09-18T10:31:12.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:31:12.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:31:12.935 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:12.930+0000 7fb5adef7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:12.949 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:12.942+0000 7fb5adef7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:12.950 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:12.942+0000 7fb5adef7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:31:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:31:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:31:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:31:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:31:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:13.290 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:31:13.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:31:13.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:13.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:13.493 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:13.486+0000 7fb5adef7540 -1 Falling back to public interface 2024-09-18T10:31:13.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:14.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:14.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:14.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:31:14.661 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:31:14.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:14.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:15.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:31:16.044 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:31:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:16.144 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:16.138+0000 7fb5adef7540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:31:16.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:17.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:17.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:17.507 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:31:17.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:31:17.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:17.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:17.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:18.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:18.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:18.927 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:31:18.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:31:18.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:18.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:19.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:20.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:20.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:20.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:31:20.331 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:31:20.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:20.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:20.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:21.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:21.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:21.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:31:21.833 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:31:21.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:21.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:22.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:23.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:23.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:23.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:31:23.262 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:31:23.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:23.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:23.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:24.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:24.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:24.854 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:31:24.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:31:24.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:24.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:25.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:26.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:26.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:26.279 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:31:26.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:31:26.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:26.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:26.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:27.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:27.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:27.797 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:31:27.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:31:27.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:27.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:28.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:29.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:29.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:29.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:31:29.332 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:31:29.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:29.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:30.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:31.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:31.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:31.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:31:31.113 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:31:31.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:31.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:31.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:32.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:32.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:32.609 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:31:32.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:31:32.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:32.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:33.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:34.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:34.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:34.628 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:31:34.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:31:34.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:34.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:35.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:36.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:36.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:36.349 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:31:36.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:31:36.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:36.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:36.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:37.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:37.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:37.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:31:37.764 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:31:37.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:37.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:38.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:39.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:39.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:39.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:31:39.203 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:31:39.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:39.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:39.650 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28755,v1:127.0.0.1:6803/28755] [v2:127.0.0.1:6804/28755,v1:127.0.0.1:6805/28755] exists,up 96e56ad6-4ed7-4ee4-aa35-49e67cad47b5 2024-09-18T10:31:39.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:31:39.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:31:39.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:31:39.652 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-18T10:31:39.652 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:31:39.652 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:31:39.652 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:39.653 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:39.653 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:39.653 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:31:39.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:31:39.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:31:39.741 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:31:39.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:31:39.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:31:39.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:31:39.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:39.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:31:39.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:31:39.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:31:39.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:31:39.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:31:39.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:31:39.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:31:39.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:31:39.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:31:39.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:31:39.873 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:31:39.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:31:39.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:31:39.876 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:31:39.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:39.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:31:39.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:31:39.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:39.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:31:39.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:31:39.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:31:39.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:31:39.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:39.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:39.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:31:39.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:31:39.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:31:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:31:39.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:31:39.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:31:39.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:31:39.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:31:39.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:31:39.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:31:39.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:31:39.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:39.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:31:39.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:31:39.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:31:39.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:31:39.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:31:39.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:31:39.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:31:39.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:39.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:39.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:31:39.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:31:39.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:31:39.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:31:39.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:31:39.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:39.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:39.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:31:39.921 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:31:39.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:31:39.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:31:39.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:31:39.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-09-18T10:31:39.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:39.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:31:39.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:31:39.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:31:39.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:31:39.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-09-18T10:31:40.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:31:40.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:31:40.275 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:31:40.276 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:31:40.276 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:40.276 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:40.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:40.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-09-18T10:31:40.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:31:40.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:31:40.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:31:40.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:31:40.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:31:40.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:31:40.542 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:31:40.542 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:31:40.542 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:31:40.543 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:40.543 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:40.543 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:40.543 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:31:40.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:31:40.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:31:40.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:31:40.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:31:40.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:31:40.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:31:40.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:40.637 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:31:40.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:31:40.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:31:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:31:41.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:31:41.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:31:41.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:31:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:31:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:41.623 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:41.624 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:31:41.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:31:41.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-09-18T10:31:41.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:31:41.658 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:31:41.658 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:31:41.658 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:41.658 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:41.659 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:31:41.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:31:41.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:31:41.748 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:31:41.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-09-18T10:31:41.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:31:41.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:31:41.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:31:41.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:31:41.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:31:41.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:31:41.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:31:41.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:31:41.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b819262-429d-4de0-9947-c0955973e96d 2024-09-18T10:31:41.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2b819262-429d-4de0-9947-c0955973e96d' 2024-09-18T10:31:41.764 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 2b819262-429d-4de0-9947-c0955973e96d 2024-09-18T10:31:41.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:31:41.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANrOpmbaaRLhAABMvnSHfHwe4I2N+jJ9DkKg== 2024-09-18T10:31:41.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANrOpmbaaRLhAABMvnSHfHwe4I2N+jJ9DkKg=="}' 2024-09-18T10:31:41.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b819262-429d-4de0-9947-c0955973e96d -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:31:42.134 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:31:42.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:31:42.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQANrOpmbaaRLhAABMvnSHfHwe4I2N+jJ9DkKg== --osd-uuid 2b819262-429d-4de0-9947-c0955973e96d 2024-09-18T10:31:42.181 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.174+0000 7f7e0e189540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:42.184 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.178+0000 7f7e0e189540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:42.186 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.178+0000 7f7e0e189540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:42.188 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.182+0000 7f7e0e189540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:31:42.188 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.182+0000 7f7e0e189540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:31:42.189 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.182+0000 7f7e0e189540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:31:42.236 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:42.230+0000 7f7e0e189540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:31:48.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:31:48.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:31:48.162 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:31:48.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:31:48.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:31:48.717 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:31:48.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:31:48.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-09-18T10:31:48.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:31:48.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:31:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:31:48.747 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:48.738+0000 7f9e4038c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:48.762 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:48.754+0000 7f9e4038c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:48.763 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:48.758+0000 7f9e4038c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:31:49.174 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:31:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:49.338 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:49.330+0000 7f9e4038c540 -1 Falling back to public interface 2024-09-18T10:31:49.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:50.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:50.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:50.616 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:31:50.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:31:50.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:50.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:52.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:52.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:52.022 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:31:52.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:31:52.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:52.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:52.055 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:31:52.045+0000 7f9e4038c540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:31:52.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:53.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:53.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:53.440 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:31:53.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:31:53.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:53.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:53.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:54.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:54.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:54.876 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:31:54.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:31:54.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:54.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:55.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:56.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:56.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:56.319 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:31:56.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:31:56.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:56.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:56.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:57.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:57.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:57.877 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:31:57.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:31:57.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:57.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:31:59.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:31:59.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:31:59.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:31:59.319 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:31:59.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:31:59.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:31:59.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:00.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:00.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:00.786 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:32:00.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:32:00.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:00.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:01.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:02.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:02.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:02.214 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:32:02.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:32:02.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:02.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:02.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:03.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:03.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:03.787 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:32:03.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:32:03.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:03.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:04.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:05.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:05.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:05.539 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:32:05.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:32:05.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:05.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:06.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:07.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:07.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:07.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:32:07.469 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:32:07.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:07.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:08.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:09.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:09.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:09.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:32:09.536 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:32:09.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:09.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:10.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:11.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:11.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:11.377 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:32:11.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:32:11.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:11.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:12.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:13.073 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:32:13.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:32:13.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:13.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:13.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:14.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:14.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:14.724 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:32:14.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:32:14.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:14.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:15.187 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30835,v1:127.0.0.1:6803/30835] [v2:127.0.0.1:6804/30835,v1:127.0.0.1:6805/30835] exists,up 2b819262-429d-4de0-9947-c0955973e96d 2024-09-18T10:32:15.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:32:15.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:32:15.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:32:15.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-09-18T10:32:15.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:32:15.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:32:15.189 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:15.189 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:15.189 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:15.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:32:15.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-09-18T10:32:15.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:32:15.269 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:32:15.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:32:15.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:32:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:32:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:32:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:32:15.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:32:15.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:32:15.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:32:15.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:32:15.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:32:15.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:32:15.599 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:32:15.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:32:15.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:32:15.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:32:15.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:32:15.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:32:15.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:15.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:32:15.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:32:15.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:15.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:32:15.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:32:15.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:32:15.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:32:15.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:15.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:15.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:32:15.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:32:15.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:32:15.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:32:15.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:32:15.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:32:15.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:32:15.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:32:15.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:32:15.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:32:15.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:15.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:32:15.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:32:15.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:15.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:32:15.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:32:15.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:32:15.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:32:15.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:15.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:15.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:32:15.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:32:15.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:32:15.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:32:15.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:32:15.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:15.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:15.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:32:15.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:32:15.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:32:15.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:32:15.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:32:15.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-09-18T10:32:15.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:15.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:32:15.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:32:15.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:32:15.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:32:15.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-09-18T10:32:16.009 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:32:16.009 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:32:16.009 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:32:16.009 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:32:16.009 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:16.009 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:16.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:16.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T10:32:16.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:32:16.248 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:32:16.248 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:32:16.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:32:16.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:32:16.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:32:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:32:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:32:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:32:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:32:16.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:32:16.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:32:16.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:32:16.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:32:16.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:32:16.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:32:16.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:32:16.341 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:32:16.342 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:32:16.342 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:32:16.342 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:16.342 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:16.342 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:16.343 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:32:16.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:32:16.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:32:16.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-09-18T10:32:16.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:32:16.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:32:16.428 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:16.428 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:16.428 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:16.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:32:16.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-18T10:32:16.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:32:16.505 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:32:16.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:32:16.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:16.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:32:16.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:32:16.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:32:16.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:32:16.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:32:17.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:32:17.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:32:17.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:32:17.414 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:32:17.414 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:17.414 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:17.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:17.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:32:17.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:32:17.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:32:17.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:32:17.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:32:17.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:32:17.452 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:32:17.452 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:32:17.452 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:32:17.454 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:32:17.454 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:17.454 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:17.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:17.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:17.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:32:17.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:32:17.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:32:17.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=023a613a-c760-483b-904b-6d9320d3eb42 2024-09-18T10:32:17.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 023a613a-c760-483b-904b-6d9320d3eb42' 2024-09-18T10:32:17.457 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 023a613a-c760-483b-904b-6d9320d3eb42 2024-09-18T10:32:17.458 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:32:17.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxrOpmBSEiHBAAKXw1cj9574xLd/Lsgg46ag== 2024-09-18T10:32:17.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxrOpmBSEiHBAAKXw1cj9574xLd/Lsgg46ag=="}' 2024-09-18T10:32:17.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 023a613a-c760-483b-904b-6d9320d3eb42 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:32:17.842 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:32:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:32:17.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxrOpmBSEiHBAAKXw1cj9574xLd/Lsgg46ag== --osd-uuid 023a613a-c760-483b-904b-6d9320d3eb42 2024-09-18T10:32:17.888 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.881+0000 7f416db87540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:17.916 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.909+0000 7f416db87540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:17.918 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.909+0000 7f416db87540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:17.919 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.909+0000 7f416db87540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:32:17.919 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.909+0000 7f416db87540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:32:17.919 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.909+0000 7f416db87540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:32:18.007 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:17.997+0000 7f416db87540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:32:23.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:32:23.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:32:23.959 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:32:23.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:32:23.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:32:24.506 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:32:24.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:32:24.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:32:24.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:32:24.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:32:24.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:32:24.534 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:24.525+0000 7f61946d4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:24.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:24.553+0000 7f61946d4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:24.564 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:24.557+0000 7f61946d4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:25.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:32:25.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:32:25.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:32:25.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:32:25.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:32:25.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:25.083 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:32:25.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:32:25.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:25.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:25.138 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:25.129+0000 7f61946d4540 -1 Falling back to public interface 2024-09-18T10:32:25.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:26.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:26.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:26.526 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:32:26.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:32:26.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:26.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:26.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:27.850 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:27.841+0000 7f61946d4540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:32:27.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:27.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:27.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:32:27.975 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:32:27.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:27.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:28.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:29.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:29.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:29.537 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:32:29.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:32:29.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:29.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:29.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:30.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:30.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:30.996 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:32:30.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:32:30.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:30.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:31.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:32.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:32.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:32.405 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:32:32.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:32:32.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:32.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:32.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:33.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:33.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:33.891 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:32:33.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:32:33.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:33.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:34.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:35.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:35.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:35.299 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:32:35.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:32:35.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:35.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:35.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:36.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:36.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:36.842 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:32:36.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:32:36.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:36.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:37.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:38.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:38.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:38.265 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:32:38.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:32:38.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:38.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:38.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:39.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:39.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:39.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:32:39.806 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:32:39.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:39.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:40.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:41.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:41.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:41.208 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:32:41.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:32:41.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:41.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:41.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:42.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:42.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:42.641 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:32:42.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:32:42.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:42.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:43.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:44.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:44.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:44.296 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:32:44.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:32:44.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:44.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:45.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:46.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:46.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:46.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:32:46.505 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:32:46.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:46.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:47.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:48.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:48.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:48.277 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:32:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:32:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:49.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:32:50.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:32:50.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:32:50.349 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:32:50.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:32:50.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:32:50.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:32:50.780 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/32900,v1:127.0.0.1:6803/32900] [v2:127.0.0.1:6804/32900,v1:127.0.0.1:6805/32900] exists,up 023a613a-c760-483b-904b-6d9320d3eb42 2024-09-18T10:32:50.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:32:50.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:32:50.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:32:50.781 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-09-18T10:32:50.781 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:32:50.781 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:32:50.782 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:50.782 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:50.782 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:50.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:32:50.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-09-18T10:32:50.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:32:50.862 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:32:50.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:32:50.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:32:50.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-09-18T10:32:50.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:50.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:32:50.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:32:50.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:32:50.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:32:50.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:32:50.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:32:50.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:32:50.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:32:50.993 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:32:50.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:32:50.995 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:32:50.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:32:50.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:32:50.998 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:32:51.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:51.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:32:51.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:32:51.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:51.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:32:51.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:32:51.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:32:51.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:32:51.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:51.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:51.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:32:51.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:32:51.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:32:51.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:32:51.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:32:51.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:32:51.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:32:51.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:32:51.022 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:32:51.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:32:51.022 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:32:51.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:32:51.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:32:51.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:32:51.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:51.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:32:51.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:32:51.026 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:32:51.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:32:51.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:32:51.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:32:51.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:32:51.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:51.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:51.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:32:51.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:32:51.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:32:51.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:32:51.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:32:51.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:51.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:51.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:32:51.038 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:32:51.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:32:51.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-09-18T10:32:51.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:32:51.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:32:51.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:32:51.420 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:32:51.420 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:51.420 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:51.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:51.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-09-18T10:32:51.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:32:51.692 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:32:51.692 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:32:51.692 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:32:51.692 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:32:51.692 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:32:51.693 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:32:51.693 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:32:51.693 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:32:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:51.694 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:32:51.694 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:32:51.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:32:51.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:32:51.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:32:51.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:32:51.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:32:51.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:32:51.787 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:32:51.788 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:32:51.788 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:32:51.788 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:51.788 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:51.788 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:51.789 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:32:51.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:32:51.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:32:51.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:32:51.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:51.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:32:51.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:32:51.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:32:51.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:32:51.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:32:52.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:32:52.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:32:52.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:32:52.786 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:32:52.786 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:52.786 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:52.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:52.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:32:52.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:32:52.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-18T10:32:52.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:32:52.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:32:52.899 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:32:52.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:32:52.910 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:32:52.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1416e469-2026-4265-8116-5883a2b92735 2024-09-18T10:32:52.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1416e469-2026-4265-8116-5883a2b92735' 2024-09-18T10:32:52.914 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 1416e469-2026-4265-8116-5883a2b92735 2024-09-18T10:32:52.914 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:32:52.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBUrOpmcc97NxAA9WvWeRwpAdlHota+LVrvjg== 2024-09-18T10:32:52.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBUrOpmcc97NxAA9WvWeRwpAdlHota+LVrvjg=="}' 2024-09-18T10:32:52.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1416e469-2026-4265-8116-5883a2b92735 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:32:53.287 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:32:53.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:32:53.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQBUrOpmcc97NxAA9WvWeRwpAdlHota+LVrvjg== --osd-uuid 1416e469-2026-4265-8116-5883a2b92735 2024-09-18T10:32:53.330 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.324+0000 7efedc30b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:53.333 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.328+0000 7efedc30b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:53.335 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.328+0000 7efedc30b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:32:53.335 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.328+0000 7efedc30b540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:32:53.335 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.328+0000 7efedc30b540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:32:53.335 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.328+0000 7efedc30b540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:32:53.408 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:32:53.404+0000 7efedc30b540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:32:59.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:32:59.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:32:59.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:32:59.764 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:32:59.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:33:00.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:33:00.328 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:33:00.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:33:00.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-09-18T10:33:00.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:33:00.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:33:00.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:00.352+0000 7fb602d6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:00.358 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:00.352+0000 7fb602d6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:00.360 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:00.356+0000 7fb602d6b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:33:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:33:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:33:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:33:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:33:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:00.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:33:00.750 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:33:00.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:00.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:00.910 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:00.904+0000 7fb602d6b540 -1 Falling back to public interface 2024-09-18T10:33:01.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:02.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:02.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:02.219 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:33:02.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:33:02.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:02.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:02.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:03.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:03.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:03.633 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:33:03.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:33:03.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:03.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:03.661 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:03.656+0000 7fb602d6b540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:33:04.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:05.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:05.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:05.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:33:05.062 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:33:05.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:05.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:05.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:06.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:06.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:06.502 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:33:06.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:33:06.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:06.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:06.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:07.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:07.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:07.934 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:33:07.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:33:07.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:07.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:08.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:09.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:09.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:09.458 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:33:09.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:33:09.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:09.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:09.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:10.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:10.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:10.904 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:33:10.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:33:10.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:10.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:11.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:12.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:12.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:12.342 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:33:12.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:33:12.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:12.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:13.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:13.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:13.766 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:33:13.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:33:13.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:13.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:14.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:15.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:15.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:15.173 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:33:15.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:33:15.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:15.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:15.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:16.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:16.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:16.985 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:33:16.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:33:16.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:16.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:19.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:19.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:19.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:33:19.079 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:33:19.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:19.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:20.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:21.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:21.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:21.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:33:21.202 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:33:21.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:21.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:22.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:23.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:23.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:23.229 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:33:23.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:33:23.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:23.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:23.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:24.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:24.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:24.672 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:33:24.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:33:24.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:24.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:25.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:26.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:26.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:26.244 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:33:26.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:33:26.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:26.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:26.690 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34954,v1:127.0.0.1:6803/34954] [v2:127.0.0.1:6804/34954,v1:127.0.0.1:6805/34954] exists,up 1416e469-2026-4265-8116-5883a2b92735 2024-09-18T10:33:26.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:33:26.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:33:26.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:33:26.691 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-18T10:33:26.691 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:33:26.691 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:33:26.692 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:33:26.692 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:26.692 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:26.693 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:33:26.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:33:26.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:33:26.778 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:33:26.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:33:26.788 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:32:51.448+0000 7fbf704c4a00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-18T10:33:26.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:33:26.790 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:33:00.908+0000 7fb602d6b540 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-18T10:33:26.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:33:26.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:33:26.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:33:26.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:33:26.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:33:26.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:33:26.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:33:26.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:33:26.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:33:26.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:33:26.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:33:26.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:33:26.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:33:26.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:33:26.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:33:26.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:33:26.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:33:26.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:33:26.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:33:26.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:33:26.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:33:26.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:33:26.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:33:26.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:33:26.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:26.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:26.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:33:26.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:33:26.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:33:26.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-18T10:33:26.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-18T10:33:26.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:33:26.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:33:26.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:33:26.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:33:26.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:33:26.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:33:26.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:33:26.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:33:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:33:26.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:33:26.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:33:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:33:26.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:33:26.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:33:26.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:33:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:33:26.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:33:26.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:33:26.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:33:26.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:26.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:26.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:33:26.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:33:26.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:33:26.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:33:26.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:33:26.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:26.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:26.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:33:26.952 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:33:26.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:33:26.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:33:26.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-09-18T10:33:27.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:33:27.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:33:27.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:33:27.337 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:33:27.337 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:27.337 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:27.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:33:27.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-09-18T10:33:27.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:33:27.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:33:27.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:33:27.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:33:27.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:27.622 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:33:27.623 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:33:27.623 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:33:27.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:33:27.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:33:27.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:33:27.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:33:27.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:33:27.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:33:27.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:33:27.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:33:27.712 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:33:27.712 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:27.712 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:27.712 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:33:27.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:33:27.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:33:27.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:33:28.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:33:28.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:33:28.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:33:28.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:33:28.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:28.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:28.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:33:28.682 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:33:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:33:28.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-18T10:33:28.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:33:28.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:33:28.714 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:33:28.714 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:28.714 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:28.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:33:28.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:33:28.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:33:28.792 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:33:28.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-09-18T10:33:28.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:33:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:33:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:33:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:33:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:33:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:33:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:33:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:33:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:33:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:33:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:33:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:33:28.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:33:28.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:33:28.809 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:33:28.809 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:33:28.809 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:33:28.809 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:33:28.810 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:33:28.810 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:33:28.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:33:28.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:33:28.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:33:28.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:33:28.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:33:28.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:33:28.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:33:28.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:33:28.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:33:28.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:33:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:33:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:33:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-09-18T10:33:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:33:28.812 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:33:28.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4363cf49-5dec-4a3e-a455-02f41e6afc89 2024-09-18T10:33:28.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4363cf49-5dec-4a3e-a455-02f41e6afc89' 2024-09-18T10:33:28.813 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 4363cf49-5dec-4a3e-a455-02f41e6afc89 2024-09-18T10:33:28.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:33:28.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4rOpmywITMRAAp+EBPUyUdMIByd0Rn7aa+Q== 2024-09-18T10:33:28.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4rOpmywITMRAAp+EBPUyUdMIByd0Rn7aa+Q=="}' 2024-09-18T10:33:28.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4363cf49-5dec-4a3e-a455-02f41e6afc89 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:33:29.173 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:33:29.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:33:29.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQB4rOpmywITMRAAp+EBPUyUdMIByd0Rn7aa+Q== --osd-uuid 4363cf49-5dec-4a3e-a455-02f41e6afc89 2024-09-18T10:33:29.217 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.211+0000 7fd4cf084540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:29.220 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.215+0000 7fd4cf084540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:29.222 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.215+0000 7fd4cf084540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:29.222 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.215+0000 7fd4cf084540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:33:29.222 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.215+0000 7fd4cf084540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:33:29.222 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.215+0000 7fd4cf084540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:33:29.305 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:29.299+0000 7fd4cf084540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:33:35.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:33:35.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:33:35.328 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:33:35.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:33:35.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:33:35.915 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:33:35.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:33:35.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-09-18T10:33:35.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:33:35.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:33:35.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:33:35.942 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:35.935+0000 7f133e9a7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:35.945 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:35.939+0000 7f133e9a7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:35.952 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:35.939+0000 7f133e9a7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:33:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:33:36.309 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:33:36.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:36.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:36.483 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:36.475+0000 7f133e9a7540 -1 Falling back to public interface 2024-09-18T10:33:36.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:37.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:37.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:37.743 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:33:37.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:33:37.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:37.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:38.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:39.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:39.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:39.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:33:39.124 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:33:39.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:39.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:39.176 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:33:39.171+0000 7f133e9a7540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:33:39.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:40.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:40.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:40.572 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:33:40.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:33:40.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:40.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:40.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:41.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:41.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:41.987 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:33:41.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:33:41.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:41.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:42.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:43.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:43.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:43.424 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:33:43.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:33:43.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:43.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:43.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:44.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:44.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:44.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:33:44.911 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:33:44.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:44.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:45.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:46.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:46.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:46.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:33:46.318 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:33:46.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:46.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:47.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:47.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:47.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:33:47.896 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:33:47.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:47.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:48.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:49.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:49.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:49.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:33:49.301 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:33:49.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:49.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:49.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:50.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:50.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:50.711 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:33:50.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:33:50.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:50.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:51.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:52.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:52.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:52.349 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:33:52.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:33:52.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:52.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:52.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:53.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:53.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:53.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:33:53.944 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:33:53.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:53.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:54.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:55.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:55.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:55.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:33:55.507 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:33:55.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:55.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:56.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:57.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:57.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:57.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:33:57.158 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:33:57.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:57.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:33:58.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:33:59.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:33:59.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:33:59.225 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:33:59.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:33:59.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:33:59.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:00.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:01.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:01.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:01.421 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:34:01.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:34:01.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:01.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:01.838 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37008,v1:127.0.0.1:6803/37008] [v2:127.0.0.1:6804/37008,v1:127.0.0.1:6805/37008] exists,up 4363cf49-5dec-4a3e-a455-02f41e6afc89 2024-09-18T10:34:01.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:34:01.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:34:01.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:34:01.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-18T10:34:01.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:34:01.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:34:01.840 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:01.840 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:01.840 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:01.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:34:01.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:34:01.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:34:01.924 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:34:01.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:34:01.937 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:33:27.363+0000 7f890c9fba00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-18T10:34:01.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:34:01.940 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:33:36.479+0000 7f133e9a7540 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-18T10:34:01.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:34:01.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:01.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:34:01.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:34:01.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:34:01.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:34:01.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:34:01.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:34:01.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:34:02.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:34:02.055 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:34:02.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:34:02.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:34:02.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:34:02.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:34:02.060 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:34:02.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:02.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:34:02.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:34:02.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:02.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:34:02.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:34:02.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:34:02.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:34:02.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:02.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:02.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:34:02.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:34:02.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:34:02.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:34:02.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:34:02.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:34:02.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:34:02.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:34:02.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:34:02.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:34:02.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:34:02.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:34:02.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:34:02.088 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:34:02.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:02.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:34:02.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:34:02.090 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:02.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:34:02.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:34:02.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:34:02.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:34:02.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:02.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:02.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:34:02.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:34:02.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:34:02.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:34:02.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:34:02.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:02.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:02.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:34:02.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:34:02.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:34:02.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:34:02.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-09-18T10:34:02.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:34:02.457 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:34:02.457 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:34:02.457 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:34:02.457 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:02.457 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:02.458 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:02.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-09-18T10:34:02.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:34:02.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:34:02.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:34:02.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:34:02.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:34:02.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:34:02.735 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:34:02.735 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:34:02.735 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:34:02.736 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:02.736 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:02.736 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:02.736 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:34:02.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:34:02.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:02.826 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:34:02.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:34:02.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:34:02.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:34:03.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:34:03.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:34:03.803 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:34:03.803 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:34:03.803 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:03.803 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:03.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:03.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:34:03.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:34:03.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-18T10:34:03.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:34:03.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:34:03.838 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:03.838 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:03.838 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:03.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:34:03.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:34:03.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:34:03.920 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:34:03.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-09-18T10:34:03.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:03.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:34:03.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:34:03.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:03.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:34:03.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:34:03.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:34:03.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:34:03.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:34:03.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:34:03.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-09-18T10:34:03.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:34:03.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:34:03.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d1a8c74-fc5a-4821-acfd-9f899740cc2e 2024-09-18T10:34:03.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3d1a8c74-fc5a-4821-acfd-9f899740cc2e' 2024-09-18T10:34:03.937 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 3d1a8c74-fc5a-4821-acfd-9f899740cc2e 2024-09-18T10:34:03.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:34:03.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbrOpmZvSpOBAAQ+NU7ZalSCfuHZLNNNza5A== 2024-09-18T10:34:03.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbrOpmZvSpOBAAQ+NU7ZalSCfuHZLNNNza5A=="}' 2024-09-18T10:34:03.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d1a8c74-fc5a-4821-acfd-9f899740cc2e -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:34:04.325 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:34:04.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:34:04.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQCbrOpmZvSpOBAAQ+NU7ZalSCfuHZLNNNza5A== --osd-uuid 3d1a8c74-fc5a-4821-acfd-9f899740cc2e 2024-09-18T10:34:04.371 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.363+0000 7fb5fb4c1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:04.373 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.367+0000 7fb5fb4c1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:04.375 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.367+0000 7fb5fb4c1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:04.375 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.367+0000 7fb5fb4c1540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:34:04.375 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.367+0000 7fb5fb4c1540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:34:04.375 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.367+0000 7fb5fb4c1540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:34:04.448 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:04.443+0000 7fb5fb4c1540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:34:10.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:34:10.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:34:10.436 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:34:10.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:34:10.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:34:10.973 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:34:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:34:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-09-18T10:34:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:34:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:34:10.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:34:11.002 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:10.995+0000 7fb788529540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:11.017 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:11.011+0000 7fb788529540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:11.018 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:11.011+0000 7fb788529540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:11.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:34:11.372 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:34:11.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:11.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:11.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:11.555+0000 7fb788529540 -1 Falling back to public interface 2024-09-18T10:34:11.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:12.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:12.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:12.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:34:12.758 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:34:12.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:12.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:13.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:13.994 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:13.986+0000 7fb788529540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:34:14.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:14.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:14.149 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:34:14.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:34:14.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:14.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:14.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:15.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:15.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:15.645 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:34:15.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:34:15.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:15.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:16.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:17.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:17.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:17.058 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:34:17.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:34:17.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:17.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:17.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:18.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:18.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:18.474 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:34:18.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:34:18.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:18.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:18.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:19.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:19.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:19.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:34:19.924 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:34:19.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:19.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:20.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:21.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:21.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:21.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:34:21.335 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:34:21.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:21.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:21.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:22.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:22.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:22.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:34:22.834 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:34:22.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:22.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:23.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:24.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:24.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:24.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:34:24.258 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:34:24.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:24.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:24.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:25.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:25.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:25.816 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:34:25.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:34:25.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:25.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:26.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:27.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:27.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:27.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:34:27.362 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:34:27.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:27.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:27.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:28.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:28.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:28.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:34:28.846 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:34:28.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:28.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:29.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:30.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:30.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:30.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:34:30.762 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:34:30.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:30.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:31.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:32.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:32.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:32.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:34:32.352 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:34:32.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:32.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:32.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:33.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:33.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:33.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:34:33.793 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:34:33.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:33.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:34.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:35.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:35.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:35.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:34:35.469 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:34:35.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:35.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:36.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:37.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:37.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:37.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:34:37.026 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:34:37.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:37.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:37.603 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/39063,v1:127.0.0.1:6803/39063] [v2:127.0.0.1:6804/39063,v1:127.0.0.1:6805/39063] exists,up 3d1a8c74-fc5a-4821-acfd-9f899740cc2e 2024-09-18T10:34:37.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:34:37.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:34:37.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:34:37.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-18T10:34:37.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:34:37.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:34:37.605 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:37.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:37.606 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:37.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:34:37.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:34:37.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:34:37.687 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:34:37.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:34:37.701 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:34:02.487+0000 7f2b7fc34a00 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-18T10:34:37.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:34:37.703 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:34:11.559+0000 7fb788529540 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-09-18T10:34:37.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:34:37.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:37.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:34:37.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:34:37.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:34:37.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:34:37.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:34:37.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:34:37.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:34:37.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:34:37.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:34:37.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:34:37.821 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:34:37.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:34:37.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:34:37.823 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:34:37.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:37.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:34:37.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:34:37.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:37.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:34:37.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:34:37.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:34:37.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:34:37.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:37.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:37.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:34:37.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:34:37.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:34:37.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:34:37.850 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:34:37.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:34:37.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:34:37.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:34:37.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:34:37.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:34:37.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:37.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:34:37.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:34:37.856 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:34:37.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:34:37.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:34:37.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:34:37.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:34:37.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:37.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:37.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:34:37.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:34:37.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:34:37.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:34:37.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:34:37.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:37.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:37.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:34:37.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:34:37.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:34:37.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:34:37.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:34:37.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-09-18T10:34:38.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:34:38.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:34:38.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:34:38.267 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:34:38.267 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:38.267 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:38.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:38.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-09-18T10:34:38.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:34:38.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:34:38.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:34:38.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:34:38.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:34:38.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:34:38.534 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:34:38.534 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:34:38.534 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:34:38.535 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:38.535 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:38.535 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:38.535 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:34:38.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:34:38.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:34:38.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:34:38.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:34:38.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:34:38.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:34:38.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:34:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:34:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:34:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:34:38.635 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:38.636 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:38.636 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:38.636 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:34:38.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:34:38.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:34:38.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:34:38.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:38.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:34:38.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:34:38.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:34:38.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:34:38.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:34:39.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:34:39.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:34:39.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:34:39.616 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:34:39.617 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:39.617 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:39.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:39.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:34:39.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:34:39.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-18T10:34:39.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:34:39.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:34:39.648 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:34:39.648 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:39.648 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:39.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:34:39.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:34:39.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:34:39.725 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:34:39.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-09-18T10:34:39.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:34:39.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:34:39.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:34:39.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:34:39.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:34:39.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:34:39.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:34:39.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:34:39.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:34:39.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:39.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:34:39.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:34:39.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:34:39.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:34:39.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:34:39.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd2f690c-7637-441c-bb3d-639c08806fa4 2024-09-18T10:34:39.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cd2f690c-7637-441c-bb3d-639c08806fa4' 2024-09-18T10:34:39.740 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 cd2f690c-7637-441c-bb3d-639c08806fa4 2024-09-18T10:34:39.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:34:39.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/rOpmIuqjLBAAK/tVA+ipbN283Dsi0ZlDpg== 2024-09-18T10:34:39.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/rOpmIuqjLBAAK/tVA+ipbN283Dsi0ZlDpg=="}' 2024-09-18T10:34:39.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd2f690c-7637-441c-bb3d-639c08806fa4 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:34:40.106 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:34:40.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:34:40.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQC/rOpmIuqjLBAAK/tVA+ipbN283Dsi0ZlDpg== --osd-uuid cd2f690c-7637-441c-bb3d-639c08806fa4 2024-09-18T10:34:40.150 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.142+0000 7f83f1e90540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:40.153 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.146+0000 7f83f1e90540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:40.154 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.146+0000 7f83f1e90540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:40.155 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.146+0000 7f83f1e90540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:34:40.155 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.146+0000 7f83f1e90540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:34:40.156 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.146+0000 7f83f1e90540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:34:40.240 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:40.230+0000 7f83f1e90540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:34:46.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:34:46.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:34:46.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:34:46.236 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:34:46.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:34:46.798 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:34:46.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:34:46.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-09-18T10:34:46.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:34:46.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:34:46.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:34:46.826 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:46.818+0000 7fc7549e7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:46.828 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:46.818+0000 7fc7549e7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:46.829 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:46.822+0000 7fc7549e7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:34:47.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:34:47.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:34:47.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:34:47.192 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:34:47.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:34:47.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:34:47.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:47.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:34:47.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:47.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:47.363 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:47.354+0000 7fc7549e7540 -1 Falling back to public interface 2024-09-18T10:34:47.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:48.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:48.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:48.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:34:48.621 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:34:48.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:48.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:49.868 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:34:49.858+0000 7fc7549e7540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:34:50.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:50.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:50.034 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:34:50.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:34:50.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:50.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:51.542 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:34:51.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:34:51.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:51.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:51.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:52.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:52.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:52.959 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:34:52.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:34:52.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:52.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:53.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:54.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:54.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:54.365 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:34:54.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:34:54.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:54.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:54.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:55.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:55.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:55.896 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:34:55.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:34:55.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:55.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:56.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:57.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:57.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:57.308 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:34:57.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:34:57.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:57.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:57.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:34:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:34:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:34:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:34:58.834 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:34:58.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:34:58.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:34:59.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:00.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:00.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:00.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:35:00.234 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:35:00.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:00.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:00.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:01.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:01.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:01.676 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:35:01.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:35:01.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:01.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:02.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:03.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:03.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:03.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:35:03.219 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:35:03.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:03.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:04.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:05.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:05.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:05.206 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:35:05.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:35:05.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:05.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:05.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:06.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:06.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:06.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:35:06.771 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:35:06.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:06.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:07.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:08.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:08.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:08.599 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:35:08.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:35:08.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:08.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:09.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:10.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:10.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:10.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:35:10.499 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:35:10.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:10.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:11.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:12.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:12.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:12.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:35:12.242 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:35:12.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:12.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:12.646 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/41156,v1:127.0.0.1:6803/41156] [v2:127.0.0.1:6804/41156,v1:127.0.0.1:6805/41156] exists,up cd2f690c-7637-441c-bb3d-639c08806fa4 2024-09-18T10:35:12.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:35:12.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:35:12.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:35:12.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-18T10:35:12.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:35:12.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:35:12.648 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:12.648 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:12.648 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:12.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:35:12.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:35:12.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:35:12.725 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:35:12.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:35:12.737 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:34:38.294+0000 7f80b42aea00 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-18T10:35:12.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:35:12.739 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:34:47.358+0000 7fc7549e7540 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-18T10:35:12.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:35:12.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:12.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:35:12.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:35:12.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:35:12.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:35:12.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:35:12.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:35:12.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:35:12.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:35:12.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:35:12.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:35:12.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:35:12.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:35:12.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:35:12.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:35:12.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:12.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:35:12.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:35:12.860 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:12.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:35:12.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:35:12.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:35:12.875 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:35:12.875 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:12.875 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:12.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:35:12.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:35:12.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:35:12.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:35:12.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:35:12.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:35:12.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:35:12.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:35:12.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:35:12.884 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:35:12.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:35:12.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:35:12.887 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:35:12.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:12.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:35:12.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:35:12.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:12.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:35:12.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:35:12.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:35:12.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:35:12.894 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:12.894 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:12.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:35:12.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:35:12.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:35:12.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:35:12.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:35:12.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:12.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:12.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:35:12.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:35:12.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:35:12.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-09-18T10:35:13.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:35:13.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:35:13.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:35:13.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:35:13.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:13.250 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:13.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:13.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-09-18T10:35:13.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:35:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:35:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:35:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:35:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:35:13.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:35:13.525 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:35:13.525 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:35:13.525 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:35:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:13.529 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:35:13.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:35:13.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:35:13.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:35:13.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:35:13.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:35:13.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:13.621 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:13.622 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:35:13.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:35:13.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:35:13.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:35:13.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:13.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:35:13.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:35:13.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:35:13.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:35:13.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:35:14.612 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:35:14.612 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:35:14.612 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:35:14.612 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:35:14.613 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:14.613 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:14.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:14.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:35:14.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:35:14.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-18T10:35:14.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:35:14.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:35:14.727 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:35:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-09-18T10:35:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:35:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:35:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:35:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:14.738 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:35:14.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:35:14.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=17690cc5-4f01-4cfc-92c5-6847aed4d636 2024-09-18T10:35:14.742 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 17690cc5-4f01-4cfc-92c5-6847aed4d636 2024-09-18T10:35:14.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 17690cc5-4f01-4cfc-92c5-6847aed4d636' 2024-09-18T10:35:14.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:35:14.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDirOpmEzEqLRAAVSeKufntaRLZovYGhenocg== 2024-09-18T10:35:14.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDirOpmEzEqLRAAVSeKufntaRLZovYGhenocg=="}' 2024-09-18T10:35:14.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 17690cc5-4f01-4cfc-92c5-6847aed4d636 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:35:15.144 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:35:15.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:35:15.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQDirOpmEzEqLRAAVSeKufntaRLZovYGhenocg== --osd-uuid 17690cc5-4f01-4cfc-92c5-6847aed4d636 2024-09-18T10:35:15.188 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.177+0000 7fd6200ca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:15.191 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.181+0000 7fd6200ca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:15.193 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.185+0000 7fd6200ca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:15.193 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.185+0000 7fd6200ca540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:35:15.193 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.185+0000 7fd6200ca540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:35:15.194 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.185+0000 7fd6200ca540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:35:15.264 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:15.253+0000 7fd6200ca540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:35:21.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:35:21.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:35:21.564 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:35:21.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:35:21.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:35:22.124 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:35:22.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:35:22.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-09-18T10:35:22.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:35:22.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:35:22.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:35:22.153 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:22.141+0000 7fa263bec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:22.155 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:22.145+0000 7fa263bec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:22.156 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:22.145+0000 7fa263bec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:22.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:35:22.563 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:35:22.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:22.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:22.735 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:22.725+0000 7fa263bec540 -1 Falling back to public interface 2024-09-18T10:35:22.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:23.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:23.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:23.948 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:35:23.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:35:23.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:23.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:24.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:25.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:25.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:25.402 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:35:25.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:35:25.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:25.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:25.421 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:25.413+0000 7fa263bec540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:35:25.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:26.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:26.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:26.814 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:35:26.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:35:26.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:26.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:27.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:28.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:28.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:28.239 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:35:28.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:35:28.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:28.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:28.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:29.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:29.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:29.630 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:35:29.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:35:29.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:29.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:30.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:31.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:31.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:31.151 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:35:31.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:35:31.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:31.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:31.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:32.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:32.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:32.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:35:32.563 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:35:32.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:32.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:33.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:34.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:34.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:34.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:35:34.146 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:35:34.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:34.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:34.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:35.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:35.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:35.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:35:35.571 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:35:35.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:35.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:36.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:37.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:37.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:37.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:35:37.145 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:35:37.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:37.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:37.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:38.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:38.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:38.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:35:38.656 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:35:38.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:38.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:39.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:40.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:40.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:40.130 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:35:40.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:35:40.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:40.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:40.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:41.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:41.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:41.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:35:41.680 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:35:41.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:41.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:42.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:43.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:43.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:43.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:35:43.504 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:35:43.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:43.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:44.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:45.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:45.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:45.445 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:35:45.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:35:45.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:45.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:45.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:46.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:46.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:46.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:35:46.855 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:35:46.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:46.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:47.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:35:48.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:35:48.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:48.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:35:48.380 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:35:48.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:48.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:48.787 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43213,v1:127.0.0.1:6803/43213] [v2:127.0.0.1:6804/43213,v1:127.0.0.1:6805/43213] exists,up 17690cc5-4f01-4cfc-92c5-6847aed4d636 2024-09-18T10:35:48.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:35:48.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:35:48.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:35:48.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-18T10:35:48.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:35:48.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:35:48.790 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:48.790 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:48.790 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:48.790 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:35:48.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:35:48.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:35:48.875 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:35:48.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:35:48.888 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:35:13.277+0000 7fbd03e2fa00 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-18T10:35:48.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:35:48.891 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:35:22.729+0000 7fa263bec540 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-18T10:35:48.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:35:48.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:48.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:35:48.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:35:48.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:35:48.893 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:35:48.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:35:48.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:35:48.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:35:49.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:35:49.007 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:35:49.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:35:49.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:35:49.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:35:49.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:35:49.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:35:49.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:49.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:35:49.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:35:49.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:49.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:35:49.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:35:49.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:35:49.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:35:49.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:49.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:49.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:49.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:35:49.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:35:49.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:35:49.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:35:49.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:35:49.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:35:49.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:35:49.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:35:49.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:35:49.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:35:49.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:35:49.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:35:49.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:35:49.045 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:35:49.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:49.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:35:49.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:35:49.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:35:49.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:35:49.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:35:49.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:35:49.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:35:49.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:49.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:49.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:35:49.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:35:49.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:35:49.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-09-18T10:35:49.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:35:49.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:49.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:49.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17190 2024-09-18T10:35:49.060 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:35:49.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:35:49.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:35:49.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-09-18T10:35:49.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-09-18T10:35:49.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:49.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:35:49.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:35:49.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:35:49.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-09-18T10:35:49.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-09-18T10:35:49.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:35:49.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:35:49.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:35:49.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:35:49.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:49.445 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:49.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:49.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-09-18T10:35:49.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:35:49.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:35:49.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:35:49.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:35:49.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:35:49.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:35:49.724 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:35:49.724 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:35:49.724 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:35:49.725 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:49.725 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:49.725 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:49.726 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:35:49.727 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:35:49.727 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get fsid 2024-09-18T10:35:49.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:35:49.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:35:49.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:35:49.815 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:35:49.815 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:35:49.815 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:35:49.816 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:35:49.816 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:35:49.816 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:49.816 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:49.816 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:49.817 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:35:49.817 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:35:49.817 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.17190/ceph-mon.a.asok config get mon_host 2024-09-18T10:35:49.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-09-18T10:35:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:35:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:35:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:35:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-09-18T10:35:49.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:35:50.809 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:35:50.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:35:50.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:35:50.810 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:35:50.810 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:50.810 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:50.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:50.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:35:50.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:35:50.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-09-18T10:35:50.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:35:50.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:35:50.848 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:35:50.848 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:50.848 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:50.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-mon.a.asok 2024-09-18T10:35:50.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:35:50.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-mon.a.asok log flush 2024-09-18T10:35:50.923 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:35:50.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-09-18T10:35:50.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:35:50.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-09-18T10:35:50.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-09-18T10:35:50.938 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:35:50.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=219281e0-f58b-4beb-b546-ec14fe49bf21 2024-09-18T10:35:50.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 219281e0-f58b-4beb-b546-ec14fe49bf21' 2024-09-18T10:35:50.938 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 219281e0-f58b-4beb-b546-ec14fe49bf21 2024-09-18T10:35:50.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:35:50.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGrepmac98OBAAVTSg7hY9rj3YMNTThhbI1w== 2024-09-18T10:35:50.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGrepmac98OBAAVTSg7hY9rj3YMNTThhbI1w=="}' 2024-09-18T10:35:50.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 219281e0-f58b-4beb-b546-ec14fe49bf21 -i td/test-erasure-code-plugins/0/new.json 2024-09-18T10:35:51.321 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:35:51.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-09-18T10:35:51.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQAGrepmac98OBAAVTSg7hY9rj3YMNTThhbI1w== --osd-uuid 219281e0-f58b-4beb-b546-ec14fe49bf21 2024-09-18T10:35:51.365 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.356+0000 7f4aff447540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:51.368 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.360+0000 7f4aff447540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:51.369 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.364+0000 7f4aff447540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:51.371 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.364+0000 7f4aff447540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:35:51.371 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.364+0000 7f4aff447540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:35:51.371 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.364+0000 7f4aff447540 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-09-18T10:35:51.445 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:51.436+0000 7f4aff447540 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-09-18T10:35:57.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-09-18T10:35:57.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:35:57.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:35:57.800 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:35:57.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:35:58.315 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:35:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:35:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81516426-e11a-4335-9cdd-86425923a9b2 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.17190/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-09-18T10:35:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:35:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:35:58.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:35:58.342 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:58.336+0000 7f681838a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:58.368 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:58.360+0000 7f681838a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:58.370 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:58.364+0000 7f681838a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:35:58.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:35:58.786 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:35:58.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:35:58.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:35:58.952 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:35:58.944+0000 7f681838a540 -1 Falling back to public interface 2024-09-18T10:35:59.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:00.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:00.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:00.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:36:00.152 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:36:00.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:00.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:00.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:01.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:01.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:01.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:36:01.559 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:36:01.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:01.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:01.582 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:01.576+0000 7f681838a540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:36:01.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:02.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:02.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:02.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:36:02.988 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:36:02.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:02.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:03.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:04.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:04.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:04.447 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:36:04.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:36:04.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:04.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:04.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:05.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:05.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:05.879 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:36:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:36:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:06.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:07.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:07.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:07.307 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:36:07.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:36:07.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:07.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:07.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:08.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:08.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:08.724 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:36:08.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:36:08.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:08.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:10.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:10.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:10.331 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:36:10.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:36:10.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:10.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:10.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:11.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:11.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:11.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:36:11.761 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:36:11.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:11.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:12.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:13.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:13.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:13.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:36:13.297 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:36:13.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:13.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:14.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:14.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:14.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:36:14.847 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:36:14.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:14.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:15.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:16.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:16.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:16.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:36:16.358 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:36:16.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:16.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:16.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:17.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:17.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:17.867 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:36:17.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:36:17.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:17.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:18.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:19.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:19.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:19.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:36:19.487 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:36:19.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:19.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:20.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:21.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:21.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:21.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:36:21.250 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:36:21.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:21.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:22.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:23.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:23.525 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:36:23.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:36:23.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:23.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:24.069 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/45293,v1:127.0.0.1:6803/45293] [v2:127.0.0.1:6804/45293,v1:127.0.0.1:6805/45293] exists,up 219281e0-f58b-4beb-b546-ec14fe49bf21 2024-09-18T10:36:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:36:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:36:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:36:24.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-09-18T10:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:36:24.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17190/ceph-osd.0.asok 2024-09-18T10:36:24.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-09-18T10:36:24.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.17190/ceph-osd.0.asok log flush 2024-09-18T10:36:24.160 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:36:24.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-09-18T10:36:24.173 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:35:49.472+0000 7f4c5c188a00 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-18T10:36:24.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-09-18T10:36:24.176 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:35:58.948+0000 7f681838a540 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-09-18T10:36:24.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-09-18T10:36:24.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:36:24.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:36:24.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:36:24.178 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:36:24.178 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:36:24.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:36:24.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:36:24.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:36:24.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:36:24.292 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:36:24.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:36:24.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:36:24.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:36:24.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:36:24.297 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:36:24.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:36:24.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:36:24.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:36:24.299 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:36:24.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:36:24.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:36:24.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:36:24.314 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:36:24.314 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:24.314 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:36:24.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:36:24.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:36:24.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:36:24.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-09-18T10:36:24.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T10:36:24.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-erasure-code-plugins 0 2024-09-18T10:36:24.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-09-18T10:36:24.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T10:36:24.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-09-18T10:36:24.318 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:36:24.318 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:36:24.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:36:24.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:36:24.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:36:24.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:36:24.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:36:24.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:36:24.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:36:24.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:36:24.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:36:24.327 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:36:24.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:36:24.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:36:24.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:36:24.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:36:24.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:36:24.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T10:36:24.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-09-18T10:36:24.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:36:24.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:24.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17190 2024-09-18T10:36:24.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17190 2024-09-18T10:36:24.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:36:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:36:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T10:36:24.338 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T10:36:24.338 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T10:36:24.352 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-09-18T10:36:24.353 DEBUG:teuthology.orchestra.run.smithi174:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-09-18T10:36:24.369 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T10:36:24.381 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T10:36:24.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-erasure-code 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-09-18T10:36:24.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-09-18T10:36:24.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none ' 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:36:24.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-18T10:36:24.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:36:24.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:36:24.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:36:24.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:36:24.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:36:24.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:36:24.389 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:36:24.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:36:24.391 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:36:24.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:36:24.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:36:24.393 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:36:24.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:36:24.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:36:24.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:36:24.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:36:24.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:36:24.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:36:24.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-18T10:36:24.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:36:24.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:24.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:24.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46250 2024-09-18T10:36:24.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:36:24.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:36:24.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-09-18T10:36:24.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:36:24.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:24.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:24.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46250 2024-09-18T10:36:24.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:36:24.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T10:36:24.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-09-18T10:36:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-09-18T10:36:24.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:36:24.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:36:24.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:36:24.811 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:36:24.811 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:24.811 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:24.812 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:36:24.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T10:36:25.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:36:25.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:36:25.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:36:25.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:36:25.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:36:25.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:36:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:36:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:36:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:36:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:36:25.096 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:25.096 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:25.096 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46250/ceph-mon.a.asok 2024-09-18T10:36:25.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:36:25.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46250/ceph-mon.a.asok config get fsid 2024-09-18T10:36:25.189 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:36:25.189 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:36:25.189 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:36:25.189 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:36:25.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:36:25.190 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:36:25.190 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:36:25.190 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:36:25.190 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:36:25.190 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:25.191 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:25.191 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46250/ceph-mon.a.asok 2024-09-18T10:36:25.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:36:25.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.46250/ceph-mon.a.asok config get mon_host 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-09-18T10:36:25.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:36:26.173 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:36:26.173 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:36:26.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:36:26.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:36:26.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:26.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:26.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:36:26.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:36:26.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:36:26.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-09-18T10:36:26.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:36:26.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:36:26.213 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:36:26.213 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:26.213 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:26.213 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46250/ceph-mon.a.asok 2024-09-18T10:36:26.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-09-18T10:36:26.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.46250/ceph-mon.a.asok log flush 2024-09-18T10:36:26.293 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:36:26.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-09-18T10:36:26.303 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:36:24.844+0000 7f2e06a62a00 0 load: jerasure load: lrc 2024-09-18T10:36:26.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-09-18T10:36:26.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:36:26.308 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:36:26.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:36:26.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:36:26.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:36:26.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:36:26.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-09-18T10:36:26.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:36:26.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=69e0c378-02f6-4032-9831-2c7755bdc5b9 2024-09-18T10:36:26.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 69e0c378-02f6-4032-9831-2c7755bdc5b9' 2024-09-18T10:36:26.311 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 69e0c378-02f6-4032-9831-2c7755bdc5b9 2024-09-18T10:36:26.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:36:26.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqrepmdUgvExAAl+bVDQ/g6RQWsaZ9zgAwvQ== 2024-09-18T10:36:26.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqrepmdUgvExAAl+bVDQ/g6RQWsaZ9zgAwvQ=="}' 2024-09-18T10:36:26.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 69e0c378-02f6-4032-9831-2c7755bdc5b9 -i td/test-erasure-code/0/new.json 2024-09-18T10:36:26.692 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:36:26.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-09-18T10:36:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqrepmdUgvExAAl+bVDQ/g6RQWsaZ9zgAwvQ== --osd-uuid 69e0c378-02f6-4032-9831-2c7755bdc5b9 2024-09-18T10:36:26.731 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.724+0000 7f875b377540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:36:26.733 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.724+0000 7f875b377540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:36:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.728+0000 7f875b377540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:36:26.736 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.728+0000 7f875b377540 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-18T10:36:26.736 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.728+0000 7f875b377540 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-18T10:36:26.736 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.728+0000 7f875b377540 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-09-18T10:36:26.795 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:26.788+0000 7f875b377540 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-09-18T10:36:33.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-09-18T10:36:33.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:36:33.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:36:33.139 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:36:33.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:36:33.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:36:33.713 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:36:33.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:36:33.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:36:33.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:36:33.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:36:33.742 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:33.735+0000 7ff4cbdf5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:36:33.755 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:33.747+0000 7ff4cbdf5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:36:33.756 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:33.747+0000 7ff4cbdf5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:36:34.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:36:34.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:36:34.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:36:34.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:36:34.186 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:36:34.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:36:34.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:34.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:36:34.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:34.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:34.312 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:34.303+0000 7ff4cbdf5540 -1 Falling back to public interface 2024-09-18T10:36:34.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:35.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:35.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:35.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:36:35.607 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:36:35.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:35.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:36.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:36.851 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:36:36.843+0000 7ff4cbdf5540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:36:37.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:37.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:37.043 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:36:37.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:36:37.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:37.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:37.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:38.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:38.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:38.514 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:36:38.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:36:38.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:38.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:38.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:39.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:39.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:39.951 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:36:39.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:36:39.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:39.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:40.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:41.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:41.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:41.380 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:36:41.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:36:41.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:41.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:41.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:42.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:42.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:42.914 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:36:42.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:36:42.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:42.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:43.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:44.344 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:36:44.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:36:44.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:44.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:44.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:45.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:45.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:45.845 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:36:45.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:36:45.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:45.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:46.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:47.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:47.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:47.246 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:36:47.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:36:47.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:47.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:47.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:48.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:48.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:48.672 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:36:48.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:36:48.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:48.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:49.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:50.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:50.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:50.250 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:36:50.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:36:50.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:50.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:50.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:51.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:51.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:51.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:36:51.750 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:36:51.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:51.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:52.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:53.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:53.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:53.263 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:36:53.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:36:53.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:53.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:54.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:55.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:55.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:55.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:36:55.145 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:36:55.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:55.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:57.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:57.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:57.180 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:36:57.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:36:57.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:57.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:58.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:36:59.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:36:59.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:36:59.323 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:36:59.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:36:59.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:36:59.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:36:59.768 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/47374,v1:127.0.0.1:6803/47374] [v2:127.0.0.1:6804/47374,v1:127.0.0.1:6805/47374] exists,up 69e0c378-02f6-4032-9831-2c7755bdc5b9 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T10:36:59.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:36:59.771 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:36:59.772 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:36:59.772 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:36:59.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:36:59.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:36:59.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:36:59.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:36:59.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:36:59.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-09-18T10:36:59.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:36:59.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4fda44a4-537d-4a6d-ada7-f1a56d2b4166 2024-09-18T10:36:59.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4fda44a4-537d-4a6d-ada7-f1a56d2b4166' 2024-09-18T10:36:59.778 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 4fda44a4-537d-4a6d-ada7-f1a56d2b4166 2024-09-18T10:36:59.778 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:36:59.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBLrepmTkMCLxAAxSFKsyOZq1e55VBdru01qA== 2024-09-18T10:36:59.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBLrepmTkMCLxAAxSFKsyOZq1e55VBdru01qA=="}' 2024-09-18T10:36:59.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4fda44a4-537d-4a6d-ada7-f1a56d2b4166 -i td/test-erasure-code/1/new.json 2024-09-18T10:37:00.334 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:37:00.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-09-18T10:37:00.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLrepmTkMCLxAAxSFKsyOZq1e55VBdru01qA== --osd-uuid 4fda44a4-537d-4a6d-ada7-f1a56d2b4166 2024-09-18T10:37:00.384 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.375+0000 7f7966812540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:00.388 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.379+0000 7f7966812540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:00.391 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.383+0000 7f7966812540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:00.392 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.383+0000 7f7966812540 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-18T10:37:00.392 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.383+0000 7f7966812540 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-18T10:37:00.392 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.383+0000 7f7966812540 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-09-18T10:37:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:00.487+0000 7f7966812540 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-09-18T10:37:06.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-09-18T10:37:06.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:37:06.737 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T10:37:06.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T10:37:06.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:37:07.275 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T10:37:07.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T10:37:07.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:37:07.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:37:07.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:37:07.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:37:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:07.295+0000 7f6416db4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:07.373 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:07.363+0000 7f6416db4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:07.375 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:07.367+0000 7f6416db4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T10:37:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:37:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:07.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:07.928 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:07.919+0000 7f6416db4540 -1 Falling back to public interface 2024-09-18T10:37:08.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:09.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:09.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:09.150 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:37:09.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:37:09.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:09.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:09.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:10.495 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:10.487+0000 7f6416db4540 -1 osd.1 0 log_to_monitors true 2024-09-18T10:37:10.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:10.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:10.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:37:10.553 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:37:10.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:10.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:11.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:12.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:12.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:12.218 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:37:12.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:37:12.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:12.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:12.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:13.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:13.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:13.743 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:37:13.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:37:13.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:13.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:14.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:15.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:15.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:15.312 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:37:15.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:37:15.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:15.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:15.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:16.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:16.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:16.751 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:37:16.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:37:16.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:16.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:17.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:18.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:18.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:18.299 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:37:18.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:37:18.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:18.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:18.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:19.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:19.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:19.777 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:37:19.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:37:19.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:19.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:20.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:21.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:21.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:21.308 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:37:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:37:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:21.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:22.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:22.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:22.742 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:37:22.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:37:22.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:22.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:23.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:24.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:24.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:24.589 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:37:24.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:37:24.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:24.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:25.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:26.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:26.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:26.102 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:37:26.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:37:26.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:26.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:26.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:27.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:27.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:27.649 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:37:27.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:37:27.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:27.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:28.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:29.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:29.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:29.308 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:37:29.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:37:29.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:29.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:30.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:31.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:31.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:31.304 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:37:31.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:37:31.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:31.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:32.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:33.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:33.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:33.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:37:33.040 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:37:33.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:33.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:33.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:34.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:34.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:34.649 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:37:34.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:37:34.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:34.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:37:35.062 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/49141,v1:127.0.0.1:6811/49141] [v2:127.0.0.1:6812/49141,v1:127.0.0.1:6813/49141] exists,up 4fda44a4-537d-4a6d-ada7-f1a56d2b4166 2024-09-18T10:37:35.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:37:35.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:37:35.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:37:35.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:37:35.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-09-18T10:37:35.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:37:35.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:37:35.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:37:35.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:37:35.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:37:35.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:37:35.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:37:35.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-09-18T10:37:35.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:37:35.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ee0f9cc-8022-4962-8d21-ed4830481a50 2024-09-18T10:37:35.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3ee0f9cc-8022-4962-8d21-ed4830481a50' 2024-09-18T10:37:35.070 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 3ee0f9cc-8022-4962-8d21-ed4830481a50 2024-09-18T10:37:35.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:37:35.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvrepmhTzNBBAAVLIxBnbzUltUVMclfst3IA== 2024-09-18T10:37:35.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvrepmhTzNBBAAVLIxBnbzUltUVMclfst3IA=="}' 2024-09-18T10:37:35.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ee0f9cc-8022-4962-8d21-ed4830481a50 -i td/test-erasure-code/2/new.json 2024-09-18T10:37:35.623 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:37:35.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-09-18T10:37:35.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvrepmhTzNBBAAVLIxBnbzUltUVMclfst3IA== --osd-uuid 3ee0f9cc-8022-4962-8d21-ed4830481a50 2024-09-18T10:37:35.669 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.658+0000 7fadf8ef7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:35.672 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.662+0000 7fadf8ef7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:35.673 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.662+0000 7fadf8ef7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:35.674 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.666+0000 7fadf8ef7540 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-18T10:37:35.674 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.666+0000 7fadf8ef7540 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-18T10:37:35.674 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.666+0000 7fadf8ef7540 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-09-18T10:37:35.786 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:35.778+0000 7fadf8ef7540 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-09-18T10:37:41.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-09-18T10:37:41.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:37:41.964 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T10:37:41.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T10:37:41.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:37:42.551 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:37:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T10:37:42.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:37:42.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:37:42.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:37:42.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:37:42.579 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:42.570+0000 7fb25e4a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:42.582 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:42.574+0000 7fb25e4a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:42.583 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:42.574+0000 7fb25e4a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:37:42.880 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:42.870+0000 7fb25e4a3540 -1 Falling back to public interface 2024-09-18T10:37:42.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T10:37:42.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:37:42.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:37:42.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:37:42.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:37:42.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:42.996 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:37:42.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:37:42.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:42.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:43.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:44.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:44.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:44.443 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:37:44.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:37:44.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:44.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:44.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:45.709 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:37:45.698+0000 7fb25e4a3540 -1 osd.2 0 log_to_monitors true 2024-09-18T10:37:45.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:45.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:45.860 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:37:45.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:37:45.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:45.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:46.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:47.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:47.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:47.391 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:37:47.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:37:47.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:47.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:47.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:48.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:48.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:48.904 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:37:48.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:37:48.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:48.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:49.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:50.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:50.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:50.306 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:37:50.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:37:50.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:50.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:50.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:51.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:51.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:51.832 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:37:51.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:37:51.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:51.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:52.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:53.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:53.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:53.303 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:37:53.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:37:53.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:53.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:53.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:54.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:54.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:54.957 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:37:54.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:37:54.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:54.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:55.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:56.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:56.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:56.562 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:37:56.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:37:56.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:56.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:56.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:57.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:57.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:57.983 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:37:57.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:37:57.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:57.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:37:58.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:37:59.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:37:59.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:37:59.609 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:37:59.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:37:59.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:37:59.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:00.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:01.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:01.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:01.217 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:38:01.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:38:01.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:01.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:01.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:02.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:02.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:02.935 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:38:02.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:38:02.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:02.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:03.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:04.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:04.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:04.506 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:38:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:38:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:05.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:06.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:06.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:06.478 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:38:06.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:38:06.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:06.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:07.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:08.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:08.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:08.063 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:38:08.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:38:08.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:08.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:08.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:09.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:09.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:09.662 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:38:09.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:38:09.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:09.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:38:10.192 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/50899,v1:127.0.0.1:6819/50899] [v2:127.0.0.1:6820/50899,v1:127.0.0.1:6821/50899] exists,up 3ee0f9cc-8022-4962-8d21-ed4830481a50 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:38:10.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:38:10.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:38:10.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:38:10.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=371b7af5-ddce-4a71-8773-3eeee9ba07d2 2024-09-18T10:38:10.201 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 371b7af5-ddce-4a71-8773-3eeee9ba07d2 2024-09-18T10:38:10.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 371b7af5-ddce-4a71-8773-3eeee9ba07d2' 2024-09-18T10:38:10.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:38:10.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSrepmfvedDBAAb4491w+a8wiSP5kOPyeSYg== 2024-09-18T10:38:10.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSrepmfvedDBAAb4491w+a8wiSP5kOPyeSYg=="}' 2024-09-18T10:38:10.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 371b7af5-ddce-4a71-8773-3eeee9ba07d2 -i td/test-erasure-code/3/new.json 2024-09-18T10:38:10.753 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:38:10.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-09-18T10:38:10.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSrepmfvedDBAAb4491w+a8wiSP5kOPyeSYg== --osd-uuid 371b7af5-ddce-4a71-8773-3eeee9ba07d2 2024-09-18T10:38:10.800 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.790+0000 7f04b426c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:10.802 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.794+0000 7f04b426c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:10.804 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.794+0000 7f04b426c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:10.805 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.794+0000 7f04b426c540 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-18T10:38:10.805 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.794+0000 7f04b426c540 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-18T10:38:10.805 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.794+0000 7f04b426c540 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-09-18T10:38:10.946 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:10.934+0000 7f04b426c540 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-09-18T10:38:16.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-09-18T10:38:16.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:38:16.989 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T10:38:16.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T10:38:16.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:38:17.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T10:38:17.528 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T10:38:17.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:38:17.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:38:17.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:38:17.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:38:17.556 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:17.545+0000 7f190e27b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:17.558 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:17.549+0000 7f190e27b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:17.562 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:17.549+0000 7f190e27b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:17.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T10:38:17.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:38:17.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T10:38:17.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:38:17.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:38:17.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:17.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:38:17.964 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:38:17.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:17.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:18.117 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:18.105+0000 7f190e27b540 -1 Falling back to public interface 2024-09-18T10:38:18.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:19.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:19.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:19.442 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:38:19.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:38:19.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:19.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:19.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:20.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:20.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:20.903 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:38:20.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:38:20.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:20.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:20.929 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:20.917+0000 7f190e27b540 -1 osd.3 0 log_to_monitors true 2024-09-18T10:38:21.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:22.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:22.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:22.417 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:38:22.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:38:22.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:22.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:22.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:23.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:23.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:23.960 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:38:23.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:38:23.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:23.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:24.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:25.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:25.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:25.380 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:38:25.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:38:25.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:25.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:25.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:26.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:26.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:26.941 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:38:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:38:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:26.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:27.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:28.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:28.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:28.367 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:38:28.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:38:28.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:28.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:29.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:30.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:30.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:30.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:38:30.011 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:38:30.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:30.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:30.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:31.553 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:38:31.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:38:31.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:31.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:32.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:33.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:33.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:33.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:38:33.096 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:38:33.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:33.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:33.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:34.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:34.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:34.964 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:38:34.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:38:34.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:34.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:35.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:36.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:36.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:36.973 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:38:36.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:38:36.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:36.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:38.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:39.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:39.019 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:38:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:38:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:39.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:39.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:40.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:40.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:40.982 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:38:40.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:38:40.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:40.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:41.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:42.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:42.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:42.518 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:38:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:38:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:43.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:44.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:44.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:44.377 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:38:44.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:38:44.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:44.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:38:44.777 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/52674,v1:127.0.0.1:6827/52674] [v2:127.0.0.1:6828/52674,v1:127.0.0.1:6829/52674] exists,up 371b7af5-ddce-4a71-8773-3eeee9ba07d2 2024-09-18T10:38:44.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:38:44.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:38:44.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:38:44.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:38:44.782 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:38:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:38:44.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-09-18T10:38:44.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:38:44.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52abb799-c1a3-4fac-ab6a-de765a982caf 2024-09-18T10:38:44.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 52abb799-c1a3-4fac-ab6a-de765a982caf' 2024-09-18T10:38:44.785 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 52abb799-c1a3-4fac-ab6a-de765a982caf 2024-09-18T10:38:44.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:38:44.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0repmO4qNLxAAl8odNN9xYRBwib49D7tz4A== 2024-09-18T10:38:44.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0repmO4qNLxAAl8odNN9xYRBwib49D7tz4A=="}' 2024-09-18T10:38:44.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52abb799-c1a3-4fac-ab6a-de765a982caf -i td/test-erasure-code/4/new.json 2024-09-18T10:38:45.309 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:38:45.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-09-18T10:38:45.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0repmO4qNLxAAl8odNN9xYRBwib49D7tz4A== --osd-uuid 52abb799-c1a3-4fac-ab6a-de765a982caf 2024-09-18T10:38:45.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.345+0000 7f39bb6b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:45.359 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.349+0000 7f39bb6b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:45.361 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.349+0000 7f39bb6b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:45.362 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.353+0000 7f39bb6b9540 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-18T10:38:45.363 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.353+0000 7f39bb6b9540 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-18T10:38:45.363 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.353+0000 7f39bb6b9540 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-09-18T10:38:45.571 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:45.561+0000 7f39bb6b9540 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-09-18T10:38:51.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-09-18T10:38:51.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:38:51.821 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T10:38:51.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T10:38:51.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:38:52.370 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T10:38:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T10:38:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:38:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:38:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:38:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:38:52.401 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:52.389+0000 7f6966916540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:52.402 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:52.393+0000 7f6966916540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:52.406 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:52.393+0000 7f6966916540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:38:52.815 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:38:52.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:52.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:38:52.955 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:52.945+0000 7f6966916540 -1 Falling back to public interface 2024-09-18T10:38:53.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:54.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:54.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:54.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:38:54.243 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:38:54.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:54.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:38:54.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:55.719 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:38:55.709+0000 7f6966916540 -1 osd.4 0 log_to_monitors true 2024-09-18T10:38:55.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:55.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:55.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:38:55.765 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:38:55.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:55.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:38:56.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:57.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:57.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:57.325 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:38:57.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:38:57.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:57.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:38:57.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:38:58.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:38:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:38:58.834 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:38:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:38:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:38:58.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:38:59.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:00.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:00.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:00.283 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:39:00.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:39:00.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:00.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:00.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:01.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:01.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:01.702 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:39:01.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:39:01.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:01.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:02.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:03.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:03.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:03.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:39:03.132 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:39:03.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:03.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:03.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:04.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:04.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:04.733 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:39:04.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:39:04.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:04.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:05.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:06.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:06.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:06.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:39:06.164 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:39:06.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:06.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:06.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:07.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:07.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:07.660 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:39:07.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:39:07.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:07.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:08.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:39:09.329 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:39:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:10.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:11.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:11.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:11.264 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:39:11.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:39:11.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:11.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:12.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:13.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:13.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:13.036 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:39:13.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:39:13.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:13.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:13.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:14.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:14.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:14.678 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:39:14.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:39:14.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:14.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:15.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:16.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:16.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:16.443 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:39:16.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:39:16.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:16.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:17.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:18.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:18.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:18.307 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:39:18.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:39:18.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:18.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:18.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:19.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:19.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:19.878 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:39:19.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:39:19.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:19.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:39:20.289 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/54423,v1:127.0.0.1:6835/54423] [v2:127.0.0.1:6836/54423,v1:127.0.0.1:6837/54423] exists,up 52abb799-c1a3-4fac-ab6a-de765a982caf 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-09-18T10:39:20.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:39:20.295 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:39:20.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:39:20.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-09-18T10:39:20.297 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:39:20.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bec68d1e-4d2f-43a4-8bd6-efe39407bbb2 2024-09-18T10:39:20.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 bec68d1e-4d2f-43a4-8bd6-efe39407bbb2' 2024-09-18T10:39:20.297 INFO:tasks.workunit.client.0.smithi174.stdout:add osd5 bec68d1e-4d2f-43a4-8bd6-efe39407bbb2 2024-09-18T10:39:20.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:39:20.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYrepmFAR9EhAAjoSOnG+jtDgHcA9GfN3ZjA== 2024-09-18T10:39:20.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYrepmFAR9EhAAjoSOnG+jtDgHcA9GfN3ZjA=="}' 2024-09-18T10:39:20.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bec68d1e-4d2f-43a4-8bd6-efe39407bbb2 -i td/test-erasure-code/5/new.json 2024-09-18T10:39:20.932 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:39:20.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-09-18T10:39:20.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDYrepmFAR9EhAAjoSOnG+jtDgHcA9GfN3ZjA== --osd-uuid bec68d1e-4d2f-43a4-8bd6-efe39407bbb2 2024-09-18T10:39:20.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:20.976+0000 7f2fc0d5d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:21.045 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:21.036+0000 7f2fc0d5d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:21.047 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:21.040+0000 7f2fc0d5d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:21.048 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:21.040+0000 7f2fc0d5d540 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-18T10:39:21.048 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:21.040+0000 7f2fc0d5d540 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-18T10:39:21.048 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:21.040+0000 7f2fc0d5d540 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-09-18T10:39:21.136 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:21.128+0000 7f2fc0d5d540 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-09-18T10:39:27.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-09-18T10:39:27.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:39:27.161 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd5 key to auth repository 2024-09-18T10:39:27.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T10:39:27.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:39:27.712 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.5 2024-09-18T10:39:27.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T10:39:27.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:39:27.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:39:27.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:39:27.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:39:27.740 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:27.732+0000 7f812266d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:27.777 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:27.768+0000 7f812266d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:27.778 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:27.772+0000 7f812266d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:28.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:39:28.155 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:39:28.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:28.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:28.352 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:28.344+0000 7f812266d540 -1 Falling back to public interface 2024-09-18T10:39:28.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:29.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:29.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:29.650 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:39:29.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:39:29.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:29.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:30.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:31.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:31.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:31.060 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:39:31.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:39:31.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:31.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:31.104 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:31.096+0000 7f812266d540 -1 osd.5 0 log_to_monitors true 2024-09-18T10:39:31.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:32.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:32.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:32.644 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:39:32.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:39:32.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:32.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:33.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:34.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:34.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:34.120 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:39:34.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:39:34.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:34.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:34.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:35.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:35.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:35.591 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:39:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:39:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:36.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:37.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:37.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:37.069 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:39:37.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:39:37.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:37.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:37.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:38.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:38.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:38.483 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:39:38.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:39:38.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:38.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:39.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:40.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:40.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:40.075 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:39:40.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:39:40.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:40.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:40.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:41.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:42.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:42.611 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:39:42.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:39:42.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:42.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:42.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:43.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:43.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:43.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:39:43.006 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:39:43.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:43.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:43.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:44.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:44.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:44.501 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:39:44.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:39:44.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:44.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:45.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:46.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:46.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:46.059 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:39:46.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:39:46.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:46.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:46.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:47.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:47.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:47.594 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:39:47.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:39:47.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:47.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:48.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:49.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:49.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:49.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:39:49.628 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:39:49.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:49.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:50.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:51.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:51.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:51.347 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:39:51.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:39:51.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:51.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:52.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:53.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:53.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:53.265 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:39:53.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:39:53.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:53.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:54.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:39:55.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:39:55.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:39:55.182 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:39:55.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:39:55.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:39:55.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:39:55.621 INFO:tasks.workunit.client.0.smithi174.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/56201,v1:127.0.0.1:6843/56201] [v2:127.0.0.1:6844/56201,v1:127.0.0.1:6845/56201] exists,up bec68d1e-4d2f-43a4-8bd6-efe39407bbb2 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T10:39:55.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:39:55.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:39:55.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:39:55.626 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:39:55.626 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:39:55.626 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:39:55.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:39:55.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-09-18T10:39:55.628 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:39:55.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c62a6324-2844-44f6-902a-d13587dd83e1 2024-09-18T10:39:55.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 c62a6324-2844-44f6-902a-d13587dd83e1' 2024-09-18T10:39:55.629 INFO:tasks.workunit.client.0.smithi174.stdout:add osd6 c62a6324-2844-44f6-902a-d13587dd83e1 2024-09-18T10:39:55.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:39:55.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7repmricOJhAARU6pw0u3z/SBg5BZkRa4ug== 2024-09-18T10:39:55.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7repmricOJhAARU6pw0u3z/SBg5BZkRa4ug=="}' 2024-09-18T10:39:55.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c62a6324-2844-44f6-902a-d13587dd83e1 -i td/test-erasure-code/6/new.json 2024-09-18T10:39:56.213 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:39:56.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-09-18T10:39:56.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7repmricOJhAARU6pw0u3z/SBg5BZkRa4ug== --osd-uuid c62a6324-2844-44f6-902a-d13587dd83e1 2024-09-18T10:39:56.260 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.252+0000 7fd3903d9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:56.264 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.256+0000 7fd3903d9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:56.266 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.260+0000 7fd3903d9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:39:56.266 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.260+0000 7fd3903d9540 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-18T10:39:56.267 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.260+0000 7fd3903d9540 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-18T10:39:56.267 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.260+0000 7fd3903d9540 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-09-18T10:39:56.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:39:56.348+0000 7fd3903d9540 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-09-18T10:40:02.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-09-18T10:40:02.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:40:02.357 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd6 key to auth repository 2024-09-18T10:40:02.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T10:40:02.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:40:02.889 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.6 2024-09-18T10:40:02.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T10:40:02.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:40:02.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:40:02.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:40:02.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:40:02.921 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:02.912+0000 7f2980d08540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:02.923 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:02.916+0000 7f2980d08540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:02.926 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:02.916+0000 7f2980d08540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:40:03.321 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:40:03.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:03.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:03.472 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:03.464+0000 7f2980d08540 -1 Falling back to public interface 2024-09-18T10:40:03.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:04.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:04.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:04.696 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:40:04.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:40:04.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:04.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:05.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:06.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:06.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:06.108 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:40:06.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:40:06.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:06.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:06.160 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:06.152+0000 7f2980d08540 -1 osd.6 0 log_to_monitors true 2024-09-18T10:40:06.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:07.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:07.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:07.518 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:40:07.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:40:07.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:07.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:07.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:08.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:08.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:08.992 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:40:08.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:40:08.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:08.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:09.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:10.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:10.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:10.431 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:40:10.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:40:10.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:10.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:11.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:12.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:12.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:12.367 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:40:12.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:40:12.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:12.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:12.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:13.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:13.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:13.428 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:40:13.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:40:13.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:13.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:13.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:14.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:14.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:14.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:40:14.982 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:40:14.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:14.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:15.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:16.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:16.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:16.397 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:40:16.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:40:16.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:16.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:16.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:17.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:17.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:17.998 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:40:17.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:40:17.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:17.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:18.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:19.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:19.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:19.417 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:40:19.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:40:19.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:19.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:20.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:21.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:21.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:21.431 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:40:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:40:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:22.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:23.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:23.110 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:40:23.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:40:23.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:23.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:23.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:24.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:24.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:24.718 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:40:24.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:40:24.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:24.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:25.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:26.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:26.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:26.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:40:26.618 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:40:26.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:26.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:27.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:28.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:28.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:28.128 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:40:28.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:40:28.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:28.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:28.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:29.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:29.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:29.572 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:40:29.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:40:29.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:29.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:40:30.121 INFO:tasks.workunit.client.0.smithi174.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/57975,v1:127.0.0.1:6851/57975] [v2:127.0.0.1:6852/57975,v1:127.0.0.1:6853/57975] exists,up c62a6324-2844-44f6-902a-d13587dd83e1 2024-09-18T10:40:30.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:40:30.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:40:30.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:40:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:40:30.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:40:30.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:40:30.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:40:30.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:40:30.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:40:30.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-09-18T10:40:30.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:40:30.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff769fc2-8b69-4ed9-b1e7-4aebb096307a 2024-09-18T10:40:30.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 ff769fc2-8b69-4ed9-b1e7-4aebb096307a' 2024-09-18T10:40:30.130 INFO:tasks.workunit.client.0.smithi174.stdout:add osd7 ff769fc2-8b69-4ed9-b1e7-4aebb096307a 2024-09-18T10:40:30.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:40:30.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAerupm0qRwCBAARbviw8ClxdrLY12RI/vnuw== 2024-09-18T10:40:30.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAerupm0qRwCBAARbviw8ClxdrLY12RI/vnuw=="}' 2024-09-18T10:40:30.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff769fc2-8b69-4ed9-b1e7-4aebb096307a -i td/test-erasure-code/7/new.json 2024-09-18T10:40:30.862 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:40:30.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-09-18T10:40:30.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAerupm0qRwCBAARbviw8ClxdrLY12RI/vnuw== --osd-uuid ff769fc2-8b69-4ed9-b1e7-4aebb096307a 2024-09-18T10:40:30.907 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.900+0000 7fb2161b4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:30.910 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.904+0000 7fb2161b4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:30.911 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.904+0000 7fb2161b4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:30.912 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.904+0000 7fb2161b4540 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-18T10:40:30.912 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.904+0000 7fb2161b4540 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-18T10:40:30.913 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.904+0000 7fb2161b4540 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-09-18T10:40:31.001 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:30.992+0000 7fb2161b4540 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-09-18T10:40:37.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-09-18T10:40:37.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:40:37.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-18T10:40:37.749 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd7 key to auth repository 2024-09-18T10:40:37.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:40:38.286 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.7 2024-09-18T10:40:38.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-18T10:40:38.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:40:38.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:40:38.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:40:38.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:40:38.315 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:38.307+0000 7ff86a7ec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:38.317 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:38.307+0000 7ff86a7ec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:38.318 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:38.311+0000 7ff86a7ec540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:40:38.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-18T10:40:38.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:40:38.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-18T10:40:38.781 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:40:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:40:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:40:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:40:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:38.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:38.888 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:38.879+0000 7ff86a7ec540 -1 Falling back to public interface 2024-09-18T10:40:39.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:40.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:40.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:40.204 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:40:40.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:40:40.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:40.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:40.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:41.489 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:40:41.479+0000 7ff86a7ec540 -1 osd.7 0 log_to_monitors true 2024-09-18T10:40:41.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:41.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:41.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:40:41.614 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:40:41.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:41.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:42.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:43.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:43.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:43.222 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:40:43.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:40:43.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:43.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:43.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:44.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:44.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:44.695 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:40:44.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:40:44.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:44.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:45.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:46.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:46.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:46.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:40:46.136 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:40:46.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:46.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:46.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:47.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:47.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:47.799 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:40:47.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:40:47.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:47.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:48.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:49.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:49.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:49.482 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:40:49.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:40:49.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:49.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:49.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:50.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:50.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:50.942 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:40:50.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:40:50.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:50.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:51.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:52.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:52.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:52.383 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:40:52.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:40:52.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:52.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:52.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:53.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:53.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:53.793 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:40:53.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:40:53.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:53.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:54.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:55.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:55.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:55.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:40:55.329 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:40:55.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:55.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:56.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:57.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:57.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:57.361 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:40:57.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:40:57.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:57.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:58.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:40:59.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:40:59.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:40:59.115 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:40:59.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:40:59.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:40:59.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:40:59.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:00.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:00.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:00.956 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:41:00.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:41:00.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:00.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:41:01.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:02.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:02.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:02.523 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:41:02.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:41:02.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:02.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:41:02.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:03.990 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:41:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:41:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:41:04.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:05.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:05.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:05.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:41:05.682 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:41:05.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:05.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T10:41:06.127 INFO:tasks.workunit.client.0.smithi174.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/59750,v1:127.0.0.1:6859/59750] [v2:127.0.0.1:6860/59750,v1:127.0.0.1:6861/59750] exists,up ff769fc2-8b69-4ed9-b1e7-4aebb096307a 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-18T10:41:06.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:41:06.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:41:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:41:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-09-18T10:41:06.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:41:06.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=557abdd6-1dee-4931-8d64-4f5c8fa1902d 2024-09-18T10:41:06.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 557abdd6-1dee-4931-8d64-4f5c8fa1902d' 2024-09-18T10:41:06.137 INFO:tasks.workunit.client.0.smithi174.stdout:add osd8 557abdd6-1dee-4931-8d64-4f5c8fa1902d 2024-09-18T10:41:06.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:41:06.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCrupmkC8fCRAAOly/r43uS2JaNQyldT9aVw== 2024-09-18T10:41:06.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCrupmkC8fCRAAOly/r43uS2JaNQyldT9aVw=="}' 2024-09-18T10:41:06.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 557abdd6-1dee-4931-8d64-4f5c8fa1902d -i td/test-erasure-code/8/new.json 2024-09-18T10:41:06.664 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:41:06.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-09-18T10:41:06.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCrupmkC8fCRAAOly/r43uS2JaNQyldT9aVw== --osd-uuid 557abdd6-1dee-4931-8d64-4f5c8fa1902d 2024-09-18T10:41:06.711 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.703+0000 7f586d6e6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:06.738 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.731+0000 7f586d6e6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:06.740 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.731+0000 7f586d6e6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:06.741 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.731+0000 7f586d6e6540 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-18T10:41:06.741 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.731+0000 7f586d6e6540 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-18T10:41:06.741 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.731+0000 7f586d6e6540 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-09-18T10:41:06.830 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:06.823+0000 7f586d6e6540 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-09-18T10:41:13.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-09-18T10:41:13.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:41:13.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-18T10:41:13.088 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd8 key to auth repository 2024-09-18T10:41:13.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:41:13.681 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.8 2024-09-18T10:41:13.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-18T10:41:13.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:41:13.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:41:13.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:41:13.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:41:13.709 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:13.699+0000 7f635a1ea540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:13.711 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:13.703+0000 7f635a1ea540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:13.712 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:13.703+0000 7f635a1ea540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:14.012 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:14.003+0000 7f635a1ea540 -1 Falling back to public interface 2024-09-18T10:41:14.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:41:14.078 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:41:14.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:14.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:14.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:15.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:15.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:15.472 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:41:15.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:41:15.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:15.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:15.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:16.843 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:16.835+0000 7f635a1ea540 -1 osd.8 0 log_to_monitors true 2024-09-18T10:41:16.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:16.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:16.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:41:16.869 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:41:16.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:16.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:17.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:18.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:18.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:18.622 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:41:18.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:41:18.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:18.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:20.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:20.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:20.153 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:41:20.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:41:20.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:20.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:20.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:21.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:21.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:21.708 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:41:21.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:41:21.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:21.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:22.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:23.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:23.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:23.130 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:41:23.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:41:23.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:23.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:23.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:24.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:24.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:24.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:41:24.743 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:41:24.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:24.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:25.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:26.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:26.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:26.159 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:41:26.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:41:26.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:26.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:26.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:27.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:27.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:27.624 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:41:27.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:41:27.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:27.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:28.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:29.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:29.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:29.032 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:41:29.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:41:29.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:29.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:29.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:30.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:30.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:30.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:41:30.530 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:41:30.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:30.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:31.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:32.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:32.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:32.023 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:41:32.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:41:32.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:32.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:32.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:33.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:33.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:33.590 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:41:33.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:41:33.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:33.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:34.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:35.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:35.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:35.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:41:35.570 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:41:35.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:35.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:36.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:37.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:37.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:37.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:41:37.668 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:41:37.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:37.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:38.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:39.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:39.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:39.672 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:41:39.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:41:39.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:39.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T10:41:40.196 INFO:tasks.workunit.client.0.smithi174.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/61526,v1:127.0.0.1:6867/61526] [v2:127.0.0.1:6868/61526,v1:127.0.0.1:6869/61526] exists,up 557abdd6-1dee-4931-8d64-4f5c8fa1902d 2024-09-18T10:41:40.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:41:40.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:41:40.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:41:40.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-18T10:41:40.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-18T10:41:40.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:41:40.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:41:40.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:41:40.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:41:40.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-09-18T10:41:40.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:41:40.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=06c7b7eb-1717-4154-8d33-fdab4951ed49 2024-09-18T10:41:40.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 06c7b7eb-1717-4154-8d33-fdab4951ed49' 2024-09-18T10:41:40.205 INFO:tasks.workunit.client.0.smithi174.stdout:add osd9 06c7b7eb-1717-4154-8d33-fdab4951ed49 2024-09-18T10:41:40.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:41:40.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkrupm58b5DBAA0gcykoSutJyFu8MkyxTGZA== 2024-09-18T10:41:40.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkrupm58b5DBAA0gcykoSutJyFu8MkyxTGZA=="}' 2024-09-18T10:41:40.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 06c7b7eb-1717-4154-8d33-fdab4951ed49 -i td/test-erasure-code/9/new.json 2024-09-18T10:41:40.784 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:41:40.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-09-18T10:41:40.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkrupm58b5DBAA0gcykoSutJyFu8MkyxTGZA== --osd-uuid 06c7b7eb-1717-4154-8d33-fdab4951ed49 2024-09-18T10:41:40.832 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.823+0000 7fb2664be540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:40.885 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.875+0000 7fb2664be540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:40.887 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.879+0000 7fb2664be540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:40.889 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.879+0000 7fb2664be540 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-18T10:41:40.889 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.879+0000 7fb2664be540 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-18T10:41:40.889 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.879+0000 7fb2664be540 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-09-18T10:41:40.962 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:40.955+0000 7fb2664be540 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-09-18T10:41:47.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-09-18T10:41:47.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:41:47.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-09-18T10:41:47.256 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd9 key to auth repository 2024-09-18T10:41:47.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:41:47.796 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.9 2024-09-18T10:41:47.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-09-18T10:41:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:41:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:41:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:41:47.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:41:47.826 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:47.819+0000 7f933ec97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:47.829 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:47.819+0000 7f933ec97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:47.830 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:47.823+0000 7f933ec97540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:48.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:41:48.255 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:41:48.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:48.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:48.371 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:48.363+0000 7f933ec97540 -1 Falling back to public interface 2024-09-18T10:41:48.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:49.657 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:41:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:41:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:50.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:50.989 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:41:50.979+0000 7f933ec97540 -1 osd.9 0 log_to_monitors true 2024-09-18T10:41:51.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:51.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:51.091 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:41:51.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:41:51.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:51.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:51.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:52.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:52.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:52.754 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:41:52.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:41:52.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:52.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:53.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:54.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:54.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:54.267 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:41:54.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:41:54.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:54.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:54.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:55.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:55.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:55.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:41:55.718 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:41:55.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:55.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:56.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:57.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:57.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:57.168 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:41:57.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:41:57.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:57.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:57.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:41:58.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:41:58.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:41:58.622 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:41:58.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:41:58.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:41:58.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:41:59.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:00.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:00.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:00.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:42:00.075 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:42:00.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:00.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:00.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:01.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:01.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:01.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:42:01.622 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:42:01.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:01.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:02.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:03.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:03.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:03.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:42:03.051 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:42:03.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:03.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:03.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:04.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:04.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:04.665 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:42:04.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:42:04.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:04.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:05.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:06.602 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:42:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:42:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:07.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:08.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:08.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:08.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:42:08.587 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:42:08.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:08.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:09.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:10.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:10.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:10.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:42:10.298 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:42:10.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:10.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:11.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:12.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:12.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:12.403 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:42:12.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:42:12.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:12.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:13.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:14.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:14.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:14.131 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:42:14.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:42:14.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:14.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:14.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:15.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:15.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:15.553 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:42:15.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:42:15.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:15.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:42:15.963 INFO:tasks.workunit.client.0.smithi174.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/63276,v1:127.0.0.1:6875/63276] [v2:127.0.0.1:6876/63276,v1:127.0.0.1:6877/63276] exists,up 06c7b7eb-1717-4154-8d33-fdab4951ed49 2024-09-18T10:42:15.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:42:15.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:42:15.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:42:15.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:42:15.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:42:15.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:42:15.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:42:15.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:42:15.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-09-18T10:42:15.970 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:42:15.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a6667d9-6625-4736-a5ea-f6e08ad36c05 2024-09-18T10:42:15.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 0a6667d9-6625-4736-a5ea-f6e08ad36c05' 2024-09-18T10:42:15.971 INFO:tasks.workunit.client.0.smithi174.stdout:add osd10 0a6667d9-6625-4736-a5ea-f6e08ad36c05 2024-09-18T10:42:15.972 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:42:15.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHrupmY2THOhAAFE3tN4okwXjhBwQcorXmYg== 2024-09-18T10:42:15.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHrupmY2THOhAAFE3tN4okwXjhBwQcorXmYg=="}' 2024-09-18T10:42:15.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a6667d9-6625-4736-a5ea-f6e08ad36c05 -i td/test-erasure-code/10/new.json 2024-09-18T10:42:16.684 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:42:16.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-09-18T10:42:16.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHrupmY2THOhAAFE3tN4okwXjhBwQcorXmYg== --osd-uuid 0a6667d9-6625-4736-a5ea-f6e08ad36c05 2024-09-18T10:42:16.728 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.719+0000 7f3413919540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:42:16.732 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.723+0000 7f3413919540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:42:16.733 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.727+0000 7f3413919540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:42:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.727+0000 7f3413919540 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-18T10:42:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.727+0000 7f3413919540 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-18T10:42:16.736 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.727+0000 7f3413919540 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-09-18T10:42:16.834 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:16.827+0000 7f3413919540 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-09-18T10:42:23.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-09-18T10:42:23.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:42:23.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-09-18T10:42:23.184 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd10 key to auth repository 2024-09-18T10:42:23.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:42:23.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-09-18T10:42:23.743 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.10 2024-09-18T10:42:23.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:42:23.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:42:23.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:42:23.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:42:23.771 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:23.763+0000 7f970c669540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:42:23.774 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:23.767+0000 7f970c669540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:42:23.775 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:23.767+0000 7f970c669540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:42:24.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-09-18T10:42:24.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:42:24.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-09-18T10:42:24.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:42:24.138 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:42:24.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:42:24.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:24.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:42:24.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:24.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:24.324 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:24.315+0000 7f970c669540 -1 Falling back to public interface 2024-09-18T10:42:24.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:25.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:25.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:25.541 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:42:25.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:42:25.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:25.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:26.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:26.893 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:42:26.883+0000 7f970c669540 -1 osd.10 0 log_to_monitors true 2024-09-18T10:42:27.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:27.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:27.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:42:27.014 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:42:27.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:27.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:27.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:28.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:28.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:28.692 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:42:28.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:42:28.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:28.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:29.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:30.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:30.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:30.111 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:42:30.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:42:30.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:30.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:30.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:31.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:31.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:31.595 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:42:31.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:42:31.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:31.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:32.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:33.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:33.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:33.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:42:33.004 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:42:33.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:33.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:33.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:34.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:34.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:34.437 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:42:34.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:42:34.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:34.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:35.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:36.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:36.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:36.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:42:36.050 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:42:36.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:36.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:36.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:37.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:37.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:37.627 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:42:37.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:42:37.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:37.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:38.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:39.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:39.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:39.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:42:39.119 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:42:39.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:39.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:39.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:40.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:40.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:40.713 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:42:40.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:42:40.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:40.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:41.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:42.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:42.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:42.434 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:42:42.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:42:42.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:42.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:43.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:44.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:44.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:44.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:42:44.059 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:42:44.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:44.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:44.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:45.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:45.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:45.706 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:42:45.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:42:45.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:45.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:46.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:47.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:47.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:47.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:42:47.270 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:42:47.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:47.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:48.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:49.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:49.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:49.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:42:49.035 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:42:49.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:49.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:49.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:42:50.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:42:50.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:42:50.852 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:42:50.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:42:50.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:42:50.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-09-18T10:42:51.331 INFO:tasks.workunit.client.0.smithi174.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/65051,v1:127.0.0.1:6883/65051] [v2:127.0.0.1:6884/65051,v1:127.0.0.1:6885/65051] exists,up 0a6667d9-6625-4736-a5ea-f6e08ad36c05 2024-09-18T10:42:51.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:42:51.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:42:51.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:42:51.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-09-18T10:42:51.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-18T10:42:52.002 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' does not exist 2024-09-18T10:42:52.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-18T10:42:52.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T10:42:52.847 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T10:42:52.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:42:53.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-18T10:42:57.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-09-18T10:42:57.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:42:57.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:42:57.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:42:57.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:42:57.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:42:57.097 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:42:57.097 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:42:57.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:42:57.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:42:57.098 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:42:57.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:42:57.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:42:57.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:42:57.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:42:57.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:42:57.708 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:42:57.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:42:57.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:57.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:42:57.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836558 2024-09-18T10:42:57.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836558 2024-09-18T10:42:57.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558' 2024-09-18T10:42:57.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:57.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:42:58.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673037 2024-09-18T10:42:58.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673037 2024-09-18T10:42:58.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037' 2024-09-18T10:42:58.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:42:58.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509507 2024-09-18T10:42:58.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509507 2024-09-18T10:42:58.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507' 2024-09-18T10:42:58.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:42:58.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345974 2024-09-18T10:42:58.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345974 2024-09-18T10:42:58.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974' 2024-09-18T10:42:58.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.369 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:42:58.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182457 2024-09-18T10:42:58.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182457 2024-09-18T10:42:58.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457' 2024-09-18T10:42:58.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:42:58.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018928 2024-09-18T10:42:58.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018928 2024-09-18T10:42:58.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457 5-128849018928' 2024-09-18T10:42:58.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:42:58.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855402 2024-09-18T10:42:58.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855402 2024-09-18T10:42:58.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457 5-128849018928 6-150323855402' 2024-09-18T10:42:58.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:42:58.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691874 2024-09-18T10:42:58.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691874 2024-09-18T10:42:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457 5-128849018928 6-150323855402 7-171798691874' 2024-09-18T10:42:58.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:58.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:42:59.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528349 2024-09-18T10:42:59.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528349 2024-09-18T10:42:59.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457 5-128849018928 6-150323855402 7-171798691874 8-193273528349' 2024-09-18T10:42:59.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:59.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:42:59.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364820 2024-09-18T10:42:59.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364820 2024-09-18T10:42:59.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457 5-128849018928 6-150323855402 7-171798691874 8-193273528349 9-214748364820' 2024-09-18T10:42:59.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:42:59.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:42:59.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201294 2024-09-18T10:42:59.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201294 2024-09-18T10:42:59.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836558 1-42949673037 2-64424509507 3-85899345974 4-107374182457 5-128849018928 6-150323855402 7-171798691874 8-193273528349 9-214748364820 10-236223201294' 2024-09-18T10:42:59.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:42:59.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836558 2024-09-18T10:42:59.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:42:59.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:42:59.399 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836558 2024-09-18T10:42:59.399 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:42:59.401 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836558 2024-09-18T10:42:59.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836558 2024-09-18T10:42:59.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-09-18T10:42:59.402 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:42:59.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836558 -lt 21474836558 2024-09-18T10:42:59.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:42:59.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673037 2024-09-18T10:42:59.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:42:59.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:42:59.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673037 2024-09-18T10:42:59.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:42:59.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673037 2024-09-18T10:42:59.864 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949673037 2024-09-18T10:42:59.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-09-18T10:42:59.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:43:00.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673038 -lt 42949673037 2024-09-18T10:43:00.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:00.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509507 2024-09-18T10:43:00.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:00.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:43:00.373 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509507 2024-09-18T10:43:00.374 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:00.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509507 2024-09-18T10:43:00.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-09-18T10:43:00.376 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509507 2024-09-18T10:43:00.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:43:00.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509508 -lt 64424509507 2024-09-18T10:43:00.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:00.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345974 2024-09-18T10:43:00.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:00.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:43:00.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:00.814 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345974 2024-09-18T10:43:00.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345974 2024-09-18T10:43:00.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2024-09-18T10:43:00.815 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 85899345974 2024-09-18T10:43:00.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:43:01.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345974 -lt 85899345974 2024-09-18T10:43:01.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:01.332 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182457 2024-09-18T10:43:01.332 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:01.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:43:01.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182457 2024-09-18T10:43:01.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:01.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182457 2024-09-18T10:43:01.338 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 107374182457 2024-09-18T10:43:01.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-09-18T10:43:01.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:43:01.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182458 -lt 107374182457 2024-09-18T10:43:01.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:01.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018928 2024-09-18T10:43:01.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:01.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:43:01.766 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018928 2024-09-18T10:43:01.766 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:01.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018928 2024-09-18T10:43:01.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-09-18T10:43:01.767 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 128849018928 2024-09-18T10:43:01.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:43:02.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018929 -lt 128849018928 2024-09-18T10:43:02.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:02.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855402 2024-09-18T10:43:02.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:02.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:43:02.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855402 2024-09-18T10:43:02.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:02.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855402 2024-09-18T10:43:02.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-09-18T10:43:02.193 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 150323855402 2024-09-18T10:43:02.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:43:02.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855403 -lt 150323855402 2024-09-18T10:43:02.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:02.691 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691874 2024-09-18T10:43:02.691 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:02.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:43:02.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691874 2024-09-18T10:43:02.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:02.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691874 2024-09-18T10:43:02.696 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.7 seq 171798691874 2024-09-18T10:43:02.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-09-18T10:43:02.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:43:03.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691875 -lt 171798691874 2024-09-18T10:43:03.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:03.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528349 2024-09-18T10:43:03.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:03.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:43:03.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528349 2024-09-18T10:43:03.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:03.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528349 2024-09-18T10:43:03.147 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 193273528349 2024-09-18T10:43:03.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528349' 2024-09-18T10:43:03.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:43:03.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528350 -lt 193273528349 2024-09-18T10:43:03.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:03.625 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364820 2024-09-18T10:43:03.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:03.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:43:03.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364820 2024-09-18T10:43:03.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:03.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364820 2024-09-18T10:43:03.632 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.9 seq 214748364820 2024-09-18T10:43:03.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364820' 2024-09-18T10:43:03.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:43:04.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364821 -lt 214748364820 2024-09-18T10:43:04.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:04.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:04.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201294 2024-09-18T10:43:04.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:43:04.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201294 2024-09-18T10:43:04.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:04.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201294 2024-09-18T10:43:04.015 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.10 seq 236223201294 2024-09-18T10:43:04.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201294' 2024-09-18T10:43:04.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:43:04.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201295 -lt 236223201294 2024-09-18T10:43:04.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:43:04.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:04.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:05.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-18T10:43:05.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:43:05.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:43:05.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:43:05.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:43:05.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:43:05.006 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:43:05.006 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:43:05.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T10:43:05.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:43:05.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:05.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:05.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-18T10:43:05.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:43:05.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:43:05.956 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-09-18T10:43:05.956 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:43:05.956 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:43:05.957 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:43:05.957 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:43:05.957 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:43:05.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46250/ceph-osd.0.asok 2024-09-18T10:43:05.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-09-18T10:43:05.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.46250/ceph-osd.0.asok log flush 2024-09-18T10:43:06.039 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:43:06.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-09-18T10:43:06.054 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:36:34.311+0000 7ff4cbdf5540 0 load: jerasure load: lrc 2024-09-18T10:43:06.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-09-18T10:43:06.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-09-18T10:43:06.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-09-18T10:43:06.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-09-18T10:43:06.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-09-18T10:43:07.776 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'ecpool' already exists 2024-09-18T10:43:07.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:43:08.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-09-18T10:43:08.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:43:08.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:43:08.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:43:08.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:43:08.791 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:43:08.792 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:43:08.792 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:43:08.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:43:08.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:43:08.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:43:08.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:43:09.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:43:09.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:43:09.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:43:09.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:43:09.779 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:43:09.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:43:09.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:09.780 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:43:10.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836562 2024-09-18T10:43:10.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836562 2024-09-18T10:43:10.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562' 2024-09-18T10:43:10.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:10.111 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:43:10.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673042 2024-09-18T10:43:10.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673042 2024-09-18T10:43:10.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042' 2024-09-18T10:43:10.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:10.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:43:10.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509512 2024-09-18T10:43:10.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509512 2024-09-18T10:43:10.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512' 2024-09-18T10:43:10.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:10.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:43:11.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345979 2024-09-18T10:43:11.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345979 2024-09-18T10:43:11.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979' 2024-09-18T10:43:11.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:11.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:43:11.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182462 2024-09-18T10:43:11.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182462 2024-09-18T10:43:11.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462' 2024-09-18T10:43:11.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:11.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:43:11.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018933 2024-09-18T10:43:11.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018933 2024-09-18T10:43:11.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462 5-128849018933' 2024-09-18T10:43:11.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:11.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:43:11.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855407 2024-09-18T10:43:11.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855407 2024-09-18T10:43:11.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462 5-128849018933 6-150323855407' 2024-09-18T10:43:11.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:11.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:43:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691879 2024-09-18T10:43:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691879 2024-09-18T10:43:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462 5-128849018933 6-150323855407 7-171798691879' 2024-09-18T10:43:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:11.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:43:12.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528354 2024-09-18T10:43:12.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528354 2024-09-18T10:43:12.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462 5-128849018933 6-150323855407 7-171798691879 8-193273528354' 2024-09-18T10:43:12.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:12.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:43:12.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364825 2024-09-18T10:43:12.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364825 2024-09-18T10:43:12.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462 5-128849018933 6-150323855407 7-171798691879 8-193273528354 9-214748364825' 2024-09-18T10:43:12.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:12.225 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:43:12.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201299 2024-09-18T10:43:12.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201299 2024-09-18T10:43:12.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836562 1-42949673042 2-64424509512 3-85899345979 4-107374182462 5-128849018933 6-150323855407 7-171798691879 8-193273528354 9-214748364825 10-236223201299' 2024-09-18T10:43:12.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:12.369 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836562 2024-09-18T10:43:12.369 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:12.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:43:12.372 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836562 2024-09-18T10:43:12.373 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:12.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836562 2024-09-18T10:43:12.374 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836562 2024-09-18T10:43:12.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836562' 2024-09-18T10:43:12.375 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:43:12.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836563 -lt 21474836562 2024-09-18T10:43:12.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:12.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673042 2024-09-18T10:43:12.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:12.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:43:12.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673042 2024-09-18T10:43:12.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:12.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673042 2024-09-18T10:43:12.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-09-18T10:43:12.839 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949673042 2024-09-18T10:43:12.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:43:13.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673042 -lt 42949673042 2024-09-18T10:43:13.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:13.316 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509512 2024-09-18T10:43:13.317 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:13.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:43:13.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509512 2024-09-18T10:43:13.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:13.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509512 2024-09-18T10:43:13.322 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509512 2024-09-18T10:43:13.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-18T10:43:13.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:43:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-18T10:43:13.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:13.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:13.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345979 2024-09-18T10:43:13.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:43:13.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345979 2024-09-18T10:43:13.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:13.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345979 2024-09-18T10:43:13.849 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 85899345979 2024-09-18T10:43:13.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345979' 2024-09-18T10:43:13.850 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:43:14.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345979 -lt 85899345979 2024-09-18T10:43:14.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:14.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:14.249 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182462 2024-09-18T10:43:14.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:43:14.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182462 2024-09-18T10:43:14.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:14.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182462 2024-09-18T10:43:14.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-09-18T10:43:14.254 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 107374182462 2024-09-18T10:43:14.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:43:14.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182462 -lt 107374182462 2024-09-18T10:43:14.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:14.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018933 2024-09-18T10:43:14.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:14.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:43:14.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018933 2024-09-18T10:43:14.752 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:14.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018933 2024-09-18T10:43:14.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018933' 2024-09-18T10:43:14.754 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 128849018933 2024-09-18T10:43:14.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:43:15.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018933 -lt 128849018933 2024-09-18T10:43:15.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:15.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855407 2024-09-18T10:43:15.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:15.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:43:15.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855407 2024-09-18T10:43:15.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:15.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855407 2024-09-18T10:43:15.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-09-18T10:43:15.247 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 150323855407 2024-09-18T10:43:15.247 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:43:15.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855407 -lt 150323855407 2024-09-18T10:43:15.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:15.707 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691879 2024-09-18T10:43:15.707 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:15.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:43:15.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691879 2024-09-18T10:43:15.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:15.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691879 2024-09-18T10:43:15.713 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.7 seq 171798691879 2024-09-18T10:43:15.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-18T10:43:15.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:43:16.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691879 -lt 171798691879 2024-09-18T10:43:16.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:16.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528354 2024-09-18T10:43:16.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:16.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:43:16.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528354 2024-09-18T10:43:16.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:16.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528354 2024-09-18T10:43:16.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-09-18T10:43:16.139 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 193273528354 2024-09-18T10:43:16.140 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:43:16.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528355 -lt 193273528354 2024-09-18T10:43:16.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:16.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364825 2024-09-18T10:43:16.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:16.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:43:16.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364825 2024-09-18T10:43:16.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:16.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364825 2024-09-18T10:43:16.644 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.9 seq 214748364825 2024-09-18T10:43:16.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364825' 2024-09-18T10:43:16.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:43:17.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364825 -lt 214748364825 2024-09-18T10:43:17.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:17.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201299 2024-09-18T10:43:17.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:17.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:43:17.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201299 2024-09-18T10:43:17.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:17.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201299 2024-09-18T10:43:17.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201299' 2024-09-18T10:43:17.036 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.10 seq 236223201299 2024-09-18T10:43:17.037 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:43:17.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201299 -lt 236223201299 2024-09-18T10:43:17.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:43:17.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:17.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:18.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T10:43:18.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:43:18.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:43:18.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:43:18.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:43:18.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:43:18.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:43:18.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:43:18.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T10:43:18.528 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:43:18.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:18.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:19.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T10:43:19.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:43:19.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:43:19.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-09-18T10:43:19.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_chunk_mapping 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_put_get_isa 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_put_get_jerasure 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_put_get_lrc_advanced 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_put_get_lrc_kml 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_put_get_shec' 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-09-18T10:43:19.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_alignment_constraints: local payload=ABC 2024-09-18T10:43:19.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:250: TEST_alignment_constraints: echo ABC 2024-09-18T10:43:19.023 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-18T10:43:19.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:256: TEST_alignment_constraints: local stripe_unit=4096 2024-09-18T10:43:19.040 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-09-18T10:43:19.040 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: grep k= 2024-09-18T10:43:19.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: eval local k=2 2024-09-18T10:43:19.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:257: TEST_alignment_constraints: local k=2 2024-09-18T10:43:19.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:258: TEST_alignment_constraints: local block_size=8191 2024-09-18T10:43:19.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:259: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-09-18T10:43:19.501 INFO:tasks.workunit.client.0.smithi174.stderr:2+0 records in 2024-09-18T10:43:19.501 INFO:tasks.workunit.client.0.smithi174.stderr:2+0 records out 2024-09-18T10:43:19.501 INFO:tasks.workunit.client.0.smithi174.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.000280194 s, 58.5 MB/s 2024-09-18T10:43:19.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-09-18T10:43:19.555 INFO:tasks.workunit.client.0.smithi174.stderr:INFO: op_size has been rounded to 8192 2024-09-18T10:43:19.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:307: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:278: verify_chunk_mapping: local poolname=ecpool 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: verify_chunk_mapping: local first=0 2024-09-18T10:43:19.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:280: verify_chunk_mapping: local second=1 2024-09-18T10:43:19.978 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-18T10:43:19.979 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-18T10:43:19.999 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-18T10:43:20.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-09-18T10:43:20.000 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-18T10:43:20.001 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-18T10:43:20.016 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-18T10:43:20.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-09-18T10:43:20.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-09-18T10:43:20.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-09-18T10:43:20.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:285: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-09-18T10:43:20.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-09-18T10:43:20.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-18T10:43:20.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-18T10:43:20.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T10:43:20.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-09-18T10:43:20.223 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-18T10:43:20.224 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:43:20.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T10:43:20.634 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:43:20.634 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:43:20.634 INFO:tasks.workunit.client.0.smithi174.stderr:0' 2024-09-18T10:43:20.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-09-18T10:43:20.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: local -a osds 2024-09-18T10:43:20.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i = 0 )) 2024-09-18T10:43:20.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-18T10:43:20.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-09-18T10:43:20.734 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:43:20.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:43:20.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-18T10:43:20.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-09-18T10:43:20.837 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:43:20.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:43:20.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-18T10:43:20.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-18T10:43:20.932 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:43:20.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:43:20.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-18T10:43:20.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-09-18T10:43:21.036 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:43:21.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:43:21.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 4 )) 2024-09-18T10:43:21.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:43:21.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-18T10:43:21.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-18T10:43:21.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-09-18T10:43:21.059 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T10:43:21.059 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-09-18T10:43:21.060 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-09-18T10:43:21.060 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:43:21.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T10:43:21.461 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:43:21.461 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:43:21.461 INFO:tasks.workunit.client.0.smithi174.stderr:0' 2024-09-18T10:43:21.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: local -a osds 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-18T10:43:21.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:43:21.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:43:21.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:43:21.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:43:21.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:43:21.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-09-18T10:43:21.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-18T10:43:21.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T10:43:21.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T10:43:21.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T10:43:21.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-09-18T10:43:21.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-09-18T10:43:24.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-09-18T10:43:24.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-18T10:43:24.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:43:24.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:43:24.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:43:24.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:43:24.447 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:43:24.447 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:43:24.447 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:43:24.447 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:43:24.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:43:24.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:43:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-09-18T10:43:24.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T10:43:24.449 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.3 2024-09-18T10:43:24.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:43:24.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-09-18T10:43:24.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T10:43:24.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:43:24.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:43:24.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:43:24.481 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:24.470+0000 7f11d070c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:43:24.483 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:24.474+0000 7f11d070c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:43:24.489 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:24.474+0000 7f11d070c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:24.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:43:25.043 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:25.034+0000 7f11d070c540 -1 Falling back to public interface 2024-09-18T10:43:25.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:43:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:43:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:43:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:43:26.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:26.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:43:26.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:43:27.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:43:27.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:27.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:43:27.756 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:43:27.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:27.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:43:27.920 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:27.910+0000 7f11d070c540 -1 osd.3 73 log_to_monitors true 2024-09-18T10:43:28.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:43:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:43:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:43:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:43:29.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:29.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [20,73) [v2:127.0.0.1:6826/68293,v1:127.0.0.1:6827/68293] [v2:127.0.0.1:6828/68293,v1:127.0.0.1:6829/68293] exists,up 371b7af5-ddce-4a71-8773-3eeee9ba07d2 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:43:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:43:29.907 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:43:29.907 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:43:29.907 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:43:29.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:43:29.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:43:29.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:43:30.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:43:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:43:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:43:30.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:43:30.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:43:30.733 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:43:30.734 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:43:30.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:43:30.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:30.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:43:30.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836569 2024-09-18T10:43:30.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836569 2024-09-18T10:43:30.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569' 2024-09-18T10:43:30.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:30.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:43:31.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673048 2024-09-18T10:43:31.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673048 2024-09-18T10:43:31.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048' 2024-09-18T10:43:31.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:43:31.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509518 2024-09-18T10:43:31.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509518 2024-09-18T10:43:31.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518' 2024-09-18T10:43:31.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:43:31.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481795 2024-09-18T10:43:31.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481795 2024-09-18T10:43:31.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795' 2024-09-18T10:43:31.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:43:31.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182468 2024-09-18T10:43:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182468 2024-09-18T10:43:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468' 2024-09-18T10:43:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:43:31.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018939 2024-09-18T10:43:31.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018939 2024-09-18T10:43:31.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468 5-128849018939' 2024-09-18T10:43:31.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:43:31.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855413 2024-09-18T10:43:31.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855413 2024-09-18T10:43:31.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468 5-128849018939 6-150323855413' 2024-09-18T10:43:31.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.850 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:43:31.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691885 2024-09-18T10:43:31.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691885 2024-09-18T10:43:31.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468 5-128849018939 6-150323855413 7-171798691885' 2024-09-18T10:43:31.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:31.997 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:43:32.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528360 2024-09-18T10:43:32.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528360 2024-09-18T10:43:32.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468 5-128849018939 6-150323855413 7-171798691885 8-193273528360' 2024-09-18T10:43:32.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:32.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:43:32.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364831 2024-09-18T10:43:32.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364831 2024-09-18T10:43:32.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468 5-128849018939 6-150323855413 7-171798691885 8-193273528360 9-214748364831' 2024-09-18T10:43:32.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:32.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:43:32.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201305 2024-09-18T10:43:32.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201305 2024-09-18T10:43:32.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836569 1-42949673048 2-64424509518 3-330712481795 4-107374182468 5-128849018939 6-150323855413 7-171798691885 8-193273528360 9-214748364831 10-236223201305' 2024-09-18T10:43:32.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:32.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836569 2024-09-18T10:43:32.438 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:32.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:43:32.440 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:32.440 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836569 2024-09-18T10:43:32.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836569 2024-09-18T10:43:32.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-18T10:43:32.442 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 21474836569 2024-09-18T10:43:32.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:43:32.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-18T10:43:32.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:32.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:32.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673048 2024-09-18T10:43:32.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:43:32.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673048 2024-09-18T10:43:32.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:32.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673048 2024-09-18T10:43:32.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673048' 2024-09-18T10:43:32.906 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 42949673048 2024-09-18T10:43:32.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:43:33.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673048 -lt 42949673048 2024-09-18T10:43:33.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:33.399 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509518 2024-09-18T10:43:33.400 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:33.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:43:33.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509518 2024-09-18T10:43:33.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:33.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509518 2024-09-18T10:43:33.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-18T10:43:33.405 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 64424509518 2024-09-18T10:43:33.408 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:43:33.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509518 -lt 64424509518 2024-09-18T10:43:33.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:33.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:33.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481795 2024-09-18T10:43:33.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:43:33.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481795 2024-09-18T10:43:33.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:33.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481795 2024-09-18T10:43:33.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481795' 2024-09-18T10:43:33.874 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 330712481795 2024-09-18T10:43:33.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:43:34.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481796 -lt 330712481795 2024-09-18T10:43:34.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:34.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182468 2024-09-18T10:43:34.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:34.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:43:34.325 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182468 2024-09-18T10:43:34.325 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:34.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182468 2024-09-18T10:43:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-09-18T10:43:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.4 seq 107374182468 2024-09-18T10:43:34.328 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:43:34.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182468 -lt 107374182468 2024-09-18T10:43:34.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:34.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018939 2024-09-18T10:43:34.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:34.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:43:34.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018939 2024-09-18T10:43:34.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:34.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018939 2024-09-18T10:43:34.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018939' 2024-09-18T10:43:34.744 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.5 seq 128849018939 2024-09-18T10:43:34.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:43:35.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018939 -lt 128849018939 2024-09-18T10:43:35.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:35.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855413 2024-09-18T10:43:35.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:35.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:43:35.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855413 2024-09-18T10:43:35.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:35.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855413 2024-09-18T10:43:35.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-09-18T10:43:35.207 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.6 seq 150323855413 2024-09-18T10:43:35.208 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:43:35.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855413 -lt 150323855413 2024-09-18T10:43:35.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:35.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:35.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691885 2024-09-18T10:43:35.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:43:35.628 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691885 2024-09-18T10:43:35.628 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:35.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691885 2024-09-18T10:43:35.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-09-18T10:43:35.630 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.7 seq 171798691885 2024-09-18T10:43:35.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:43:36.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691885 -lt 171798691885 2024-09-18T10:43:36.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:36.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528360 2024-09-18T10:43:36.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:36.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:43:36.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528360 2024-09-18T10:43:36.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:36.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528360 2024-09-18T10:43:36.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-09-18T10:43:36.084 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.8 seq 193273528360 2024-09-18T10:43:36.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:43:36.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528361 -lt 193273528360 2024-09-18T10:43:36.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:36.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364831 2024-09-18T10:43:36.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:36.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:43:36.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364831 2024-09-18T10:43:36.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:36.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364831 2024-09-18T10:43:36.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364831' 2024-09-18T10:43:36.618 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.9 seq 214748364831 2024-09-18T10:43:36.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:43:37.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364831 -lt 214748364831 2024-09-18T10:43:37.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:37.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201305 2024-09-18T10:43:37.053 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:37.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:43:37.055 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201305 2024-09-18T10:43:37.055 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:37.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201305 2024-09-18T10:43:37.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201305' 2024-09-18T10:43:37.057 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.10 seq 236223201305 2024-09-18T10:43:37.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:43:37.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201305 -lt 236223201305 2024-09-18T10:43:37.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:43:37.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:37.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:38.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T10:43:38.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:43:38.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:43:38.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:43:38.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:43:38.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:43:38.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:43:38.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:43:38.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T10:43:38.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:43:38.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:38.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:39.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T10:43:39.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:43:39.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: grep --quiet SECONDecpool 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-18T10:43:39.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-18T10:43:39.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T10:43:39.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-09-18T10:43:39.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T10:43:39.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-09-18T10:43:39.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:43:39.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:43:39.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:43:39.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:43:39.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:43:39.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:43:39.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-09-18T10:43:39.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-18T10:43:39.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T10:43:39.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-09-18T10:43:39.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T10:43:39.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-09-18T10:43:39.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-09-18T10:43:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-09-18T10:43:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-18T10:43:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:43:42.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-09-18T10:43:42.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:43:42.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-09-18T10:43:42.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:43:42.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:43:42.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:43:42.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:43:42.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-09-18T10:43:42.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-09-18T10:43:42.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:43:42.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:43:42.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:43:42.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:43:42.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:43:42.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:43:42.522 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:43:42.522 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:43:42.523 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:43:42.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:43:42.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:43:42.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:43:42.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:43:42.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:43:42.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:43:42.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:43:42.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:43:42.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:43:42.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:43:42.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:43:42.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-09-18T10:43:42.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-09-18T10:43:42.525 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.5 2024-09-18T10:43:42.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:43:42.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-09-18T10:43:42.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-09-18T10:43:42.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:43:42.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:43:42.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:43:42.545 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:42.538+0000 7f10b0855540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:43:42.549 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:42.542+0000 7f10b0855540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:43:42.551 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:42.542+0000 7f10b0855540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:42.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:43:42.944 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:43:42.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:42.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:43:43.100 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:43.090+0000 7f10b0855540 -1 Falling back to public interface 2024-09-18T10:43:43.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:43:44.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:43:44.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:44.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:43:44.330 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:43:44.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:44.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:43:44.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:43:45.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:43:45.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:45.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:43:45.786 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:43:45.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:45.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:43:45.879 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:43:45.870+0000 7f10b0855540 -1 osd.5 79 log_to_monitors true 2024-09-18T10:43:46.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:43:47.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:43:47.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:43:47.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:43:47.294 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:43:47.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:43:47.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/70072,v1:127.0.0.1:6843/70072] [v2:127.0.0.1:6844/70072,v1:127.0.0.1:6845/70072] exists,up bec68d1e-4d2f-43a4-8bd6-efe39407bbb2 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:43:47.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:43:47.964 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:43:47.964 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:43:47.964 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:43:47.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:43:47.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:43:47.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:43:48.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:43:48.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:43:48.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:43:48.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:43:48.168 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:43:48.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:43:48.580 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:43:48.580 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:43:48.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:48.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:43:48.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836574 2024-09-18T10:43:48.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836574 2024-09-18T10:43:48.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574' 2024-09-18T10:43:48.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:48.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:43:49.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673054 2024-09-18T10:43:49.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673054 2024-09-18T10:43:49.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054' 2024-09-18T10:43:49.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:49.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:43:49.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509524 2024-09-18T10:43:49.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509524 2024-09-18T10:43:49.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524' 2024-09-18T10:43:49.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:49.232 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:43:49.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481801 2024-09-18T10:43:49.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481801 2024-09-18T10:43:49.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801' 2024-09-18T10:43:49.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:49.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:43:49.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182473 2024-09-18T10:43:49.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182473 2024-09-18T10:43:49.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473' 2024-09-18T10:43:49.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:49.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:43:49.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252867 2024-09-18T10:43:49.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252867 2024-09-18T10:43:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473 5-360777252867' 2024-09-18T10:43:49.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:49.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:43:49.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855418 2024-09-18T10:43:49.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855418 2024-09-18T10:43:49.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473 5-360777252867 6-150323855418' 2024-09-18T10:43:49.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:49.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:43:50.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691890 2024-09-18T10:43:50.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691890 2024-09-18T10:43:50.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473 5-360777252867 6-150323855418 7-171798691890' 2024-09-18T10:43:50.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:50.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:43:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528365 2024-09-18T10:43:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528365 2024-09-18T10:43:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473 5-360777252867 6-150323855418 7-171798691890 8-193273528365' 2024-09-18T10:43:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:50.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:43:50.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364836 2024-09-18T10:43:50.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364836 2024-09-18T10:43:50.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473 5-360777252867 6-150323855418 7-171798691890 8-193273528365 9-214748364836' 2024-09-18T10:43:50.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:43:50.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:43:50.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201310 2024-09-18T10:43:50.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201310 2024-09-18T10:43:50.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836574 1-42949673054 2-64424509524 3-330712481801 4-107374182473 5-360777252867 6-150323855418 7-171798691890 8-193273528365 9-214748364836 10-236223201310' 2024-09-18T10:43:50.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:50.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836574 2024-09-18T10:43:50.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:50.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:43:50.438 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836574 2024-09-18T10:43:50.438 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:50.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836574 2024-09-18T10:43:50.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-09-18T10:43:50.439 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 21474836574 2024-09-18T10:43:50.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:43:50.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836574 -lt 21474836574 2024-09-18T10:43:50.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:50.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:50.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673054 2024-09-18T10:43:50.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:43:50.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673054 2024-09-18T10:43:50.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:50.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673054 2024-09-18T10:43:50.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-09-18T10:43:50.847 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 42949673054 2024-09-18T10:43:50.847 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:43:51.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673054 -lt 42949673054 2024-09-18T10:43:51.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:51.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:51.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509524 2024-09-18T10:43:51.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:43:51.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509524 2024-09-18T10:43:51.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:51.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509524 2024-09-18T10:43:51.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-18T10:43:51.270 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 64424509524 2024-09-18T10:43:51.270 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:43:51.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-18T10:43:51.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:51.687 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481801 2024-09-18T10:43:51.688 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:51.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:43:51.690 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481801 2024-09-18T10:43:51.691 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:51.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481801 2024-09-18T10:43:51.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481801' 2024-09-18T10:43:51.693 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 330712481801 2024-09-18T10:43:51.693 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:43:52.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481801 -lt 330712481801 2024-09-18T10:43:52.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:52.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182473 2024-09-18T10:43:52.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:52.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:43:52.106 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182473 2024-09-18T10:43:52.106 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:52.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182473 2024-09-18T10:43:52.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-09-18T10:43:52.108 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.4 seq 107374182473 2024-09-18T10:43:52.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:43:52.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182474 -lt 107374182473 2024-09-18T10:43:52.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:52.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-360777252867 2024-09-18T10:43:52.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:52.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:43:52.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-360777252867 2024-09-18T10:43:52.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:52.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252867 2024-09-18T10:43:52.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 360777252867' 2024-09-18T10:43:52.618 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.5 seq 360777252867 2024-09-18T10:43:52.619 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:43:53.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252868 -lt 360777252867 2024-09-18T10:43:53.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:53.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-150323855418 2024-09-18T10:43:53.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:53.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:43:53.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-150323855418 2024-09-18T10:43:53.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:53.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855418 2024-09-18T10:43:53.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-09-18T10:43:53.074 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.6 seq 150323855418 2024-09-18T10:43:53.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:43:53.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855419 -lt 150323855418 2024-09-18T10:43:53.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:53.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691890 2024-09-18T10:43:53.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:53.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:43:53.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691890 2024-09-18T10:43:53.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:53.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691890 2024-09-18T10:43:53.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-18T10:43:53.593 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.7 seq 171798691890 2024-09-18T10:43:53.594 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:43:53.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691891 -lt 171798691890 2024-09-18T10:43:53.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:53.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528365 2024-09-18T10:43:53.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:53.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:43:53.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528365 2024-09-18T10:43:53.984 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:53.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528365 2024-09-18T10:43:53.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528365' 2024-09-18T10:43:53.986 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.8 seq 193273528365 2024-09-18T10:43:53.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:43:54.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528366 -lt 193273528365 2024-09-18T10:43:54.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:54.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364836 2024-09-18T10:43:54.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:54.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:43:54.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364836 2024-09-18T10:43:54.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:54.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364836 2024-09-18T10:43:54.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364836' 2024-09-18T10:43:54.467 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.9 seq 214748364836 2024-09-18T10:43:54.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:43:54.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364837 -lt 214748364836 2024-09-18T10:43:54.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:43:54.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201310 2024-09-18T10:43:54.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:43:54.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:43:54.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201310 2024-09-18T10:43:54.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:43:54.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201310 2024-09-18T10:43:54.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201310' 2024-09-18T10:43:54.911 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.10 seq 236223201310 2024-09-18T10:43:54.911 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:43:55.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201311 -lt 236223201310 2024-09-18T10:43:55.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:43:55.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:55.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:55.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-18T10:43:55.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:43:55.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:43:55.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:43:55.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:43:55.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:43:55.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:43:55.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:43:56.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-18T10:43:56.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:43:56.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:43:56.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:43:56.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-18T10:43:56.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:43:56.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:43:56.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:309: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-09-18T10:43:57.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:314: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-09-18T10:43:57.830 INFO:tasks.workunit.client.0.smithi174.stdout:crush-device-class= 2024-09-18T10:43:57.830 INFO:tasks.workunit.client.0.smithi174.stdout:crush-failure-domain=host 2024-09-18T10:43:57.830 INFO:tasks.workunit.client.0.smithi174.stdout:crush-root=default 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:k=-1 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:l=-1 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:layers=[ [ "cDD", "" ] ] 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:m=-1 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:mapping=_DD 2024-09-18T10:43:57.831 INFO:tasks.workunit.client.0.smithi174.stdout:plugin=lrc 2024-09-18T10:43:57.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:315: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-09-18T10:43:57.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-09-18T10:43:58.918 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'remap-pool' already exists 2024-09-18T10:43:58.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:43:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:323: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-09-18T10:43:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:277: verify_chunk_mapping: local dir=td/test-erasure-code 2024-09-18T10:43:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:278: verify_chunk_mapping: local poolname=remap-pool 2024-09-18T10:43:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:279: verify_chunk_mapping: local first=1 2024-09-18T10:43:59.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:280: verify_chunk_mapping: local second=2 2024-09-18T10:43:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-18T10:43:59.935 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-18T10:43:59.956 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-18T10:43:59.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-09-18T10:43:59.957 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: chunk_size 2024-09-18T10:43:59.958 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-09-18T10:43:59.980 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: chunk_size: echo 4096 2024-09-18T10:43:59.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-09-18T10:43:59.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-09-18T10:43:59.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-09-18T10:43:59.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:285: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-09-18T10:44:01.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-09-18T10:44:01.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-18T10:44:01.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-18T10:44:01.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-09-18T10:44:01.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-09-18T10:44:01.646 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-18T10:44:01.646 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:44:02.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:9' 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: local -a osds 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i = 0 )) 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-18T10:44:02.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-09-18T10:44:02.406 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:44:02.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:44:02.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-18T10:44:02.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-09-18T10:44:02.518 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:44:02.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:44:02.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-18T10:44:02.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-09-18T10:44:02.627 INFO:tasks.workunit.client.0.smithi174.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-18T10:44:02.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i++ )) 2024-09-18T10:44:02.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:288: verify_chunk_mapping: (( i < 3 )) 2024-09-18T10:44:02.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:44:02.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-09-18T10:44:02.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: osds=($(get_osds $poolname SOMETHING$poolname)) 2024-09-18T10:44:02.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-09-18T10:44:02.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-09-18T10:44:02.632 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-09-18T10:44:02.632 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-09-18T10:44:02.632 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:44:03.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-18T10:44:03.054 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:44:03.055 INFO:tasks.workunit.client.0.smithi174.stderr:9' 2024-09-18T10:44:03.055 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:294: verify_chunk_mapping: local -a osds 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-18T10:44:03.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:44:03.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:44:03.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:44:03.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-18T10:44:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T10:44:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-09-18T10:44:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T10:44:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-09-18T10:44:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:06.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-09-18T10:44:06.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:44:06.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:44:06.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:44:06.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:44:06.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:44:06.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:44:06.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:44:06.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:44:06.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:44:06.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-09-18T10:44:06.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-09-18T10:44:06.637 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.6 2024-09-18T10:44:06.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:44:06.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-09-18T10:44:06.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-09-18T10:44:06.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:44:06.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:44:06.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:44:06.670 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:06.661+0000 7f0d795a1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:44:06.674 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:06.665+0000 7f0d795a1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:44:06.681 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:06.665+0000 7f0d795a1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:44:07.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:07.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:44:07.228 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:07.217+0000 7f0d795a1540 -1 Falling back to public interface 2024-09-18T10:44:07.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:44:08.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:44:08.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:08.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:44:08.562 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:44:08.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:08.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:44:09.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:44:10.019 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:10.009+0000 7f0d795a1540 -1 osd.6 95 log_to_monitors true 2024-09-18T10:44:10.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:44:10.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:10.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:44:10.055 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:44:10.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:10.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:44:10.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:44:11.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:44:11.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:11.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:44:11.573 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:44:11.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:11.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:44:12.526 INFO:tasks.workunit.client.0.smithi174.stderr:osd.6 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [35,95) [v2:127.0.0.1:6850/72044,v1:127.0.0.1:6851/72044] [v2:127.0.0.1:6852/72044,v1:127.0.0.1:6853/72044] exists,up c62a6324-2844-44f6-902a-d13587dd83e1 2024-09-18T10:44:12.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:44:12.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:44:12.528 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:44:12.529 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:44:12.529 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:44:12.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:44:12.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:44:12.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:44:12.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:44:12.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:44:12.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:44:12.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:44:12.729 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:44:13.130 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:13.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:44:13.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836581 2024-09-18T10:44:13.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836581 2024-09-18T10:44:13.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581' 2024-09-18T10:44:13.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:13.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:44:13.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673060 2024-09-18T10:44:13.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673060 2024-09-18T10:44:13.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060' 2024-09-18T10:44:13.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:13.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:44:13.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509530 2024-09-18T10:44:13.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509530 2024-09-18T10:44:13.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530' 2024-09-18T10:44:13.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:13.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:44:14.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481808 2024-09-18T10:44:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481808 2024-09-18T10:44:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808' 2024-09-18T10:44:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.006 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:44:14.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182480 2024-09-18T10:44:14.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182480 2024-09-18T10:44:14.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480' 2024-09-18T10:44:14.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:44:14.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252874 2024-09-18T10:44:14.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252874 2024-09-18T10:44:14.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480 5-360777252874' 2024-09-18T10:44:14.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:44:14.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729603 2024-09-18T10:44:14.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729603 2024-09-18T10:44:14.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480 5-360777252874 6-429496729603' 2024-09-18T10:44:14.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:44:14.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691897 2024-09-18T10:44:14.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691897 2024-09-18T10:44:14.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480 5-360777252874 6-429496729603 7-171798691897' 2024-09-18T10:44:14.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.593 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:44:14.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528372 2024-09-18T10:44:14.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528372 2024-09-18T10:44:14.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480 5-360777252874 6-429496729603 7-171798691897 8-193273528372' 2024-09-18T10:44:14.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:44:14.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364843 2024-09-18T10:44:14.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364843 2024-09-18T10:44:14.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480 5-360777252874 6-429496729603 7-171798691897 8-193273528372 9-214748364843' 2024-09-18T10:44:14.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:14.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:44:15.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201317 2024-09-18T10:44:15.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201317 2024-09-18T10:44:15.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673060 2-64424509530 3-330712481808 4-107374182480 5-360777252874 6-429496729603 7-171798691897 8-193273528372 9-214748364843 10-236223201317' 2024-09-18T10:44:15.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:15.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836581 2024-09-18T10:44:15.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:15.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:44:15.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836581 2024-09-18T10:44:15.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:15.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836581 2024-09-18T10:44:15.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-18T10:44:15.053 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 21474836581 2024-09-18T10:44:15.053 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:44:15.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-18T10:44:15.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:15.475 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673060 2024-09-18T10:44:15.475 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:15.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:44:15.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673060 2024-09-18T10:44:15.479 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:15.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673060 2024-09-18T10:44:15.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-09-18T10:44:15.481 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 42949673060 2024-09-18T10:44:15.481 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:44:15.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673061 -lt 42949673060 2024-09-18T10:44:15.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:15.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509530 2024-09-18T10:44:15.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:15.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:44:15.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509530 2024-09-18T10:44:15.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:15.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509530 2024-09-18T10:44:15.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-18T10:44:15.905 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 64424509530 2024-09-18T10:44:15.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:44:16.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509531 -lt 64424509530 2024-09-18T10:44:16.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:16.393 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481808 2024-09-18T10:44:16.393 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:16.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:44:16.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481808 2024-09-18T10:44:16.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:16.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481808 2024-09-18T10:44:16.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481808' 2024-09-18T10:44:16.399 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 330712481808 2024-09-18T10:44:16.399 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:44:16.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481808 -lt 330712481808 2024-09-18T10:44:16.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:16.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182480 2024-09-18T10:44:16.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:16.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:44:16.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182480 2024-09-18T10:44:16.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:16.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182480 2024-09-18T10:44:16.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-09-18T10:44:16.864 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.4 seq 107374182480 2024-09-18T10:44:16.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:44:17.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182481 -lt 107374182480 2024-09-18T10:44:17.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:17.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-360777252874 2024-09-18T10:44:17.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:17.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:44:17.366 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-360777252874 2024-09-18T10:44:17.367 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:17.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252874 2024-09-18T10:44:17.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 360777252874' 2024-09-18T10:44:17.369 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.5 seq 360777252874 2024-09-18T10:44:17.369 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:44:17.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252875 -lt 360777252874 2024-09-18T10:44:17.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:17.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-429496729603 2024-09-18T10:44:17.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:17.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:44:17.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-429496729603 2024-09-18T10:44:17.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:17.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729603 2024-09-18T10:44:17.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 429496729603' 2024-09-18T10:44:17.813 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.6 seq 429496729603 2024-09-18T10:44:17.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:44:18.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729604 -lt 429496729603 2024-09-18T10:44:18.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:18.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691897 2024-09-18T10:44:18.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:18.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:44:18.270 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691897 2024-09-18T10:44:18.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691897 2024-09-18T10:44:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-09-18T10:44:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.7 seq 171798691897 2024-09-18T10:44:18.273 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:44:18.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691898 -lt 171798691897 2024-09-18T10:44:18.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:18.717 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:18.717 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528372 2024-09-18T10:44:18.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:44:18.719 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528372 2024-09-18T10:44:18.719 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:18.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528372 2024-09-18T10:44:18.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-09-18T10:44:18.722 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.8 seq 193273528372 2024-09-18T10:44:18.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:44:19.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528373 -lt 193273528372 2024-09-18T10:44:19.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:19.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-214748364843 2024-09-18T10:44:19.126 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:19.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:44:19.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-214748364843 2024-09-18T10:44:19.129 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:19.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364843 2024-09-18T10:44:19.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 214748364843' 2024-09-18T10:44:19.130 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.9 seq 214748364843 2024-09-18T10:44:19.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:44:19.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364844 -lt 214748364843 2024-09-18T10:44:19.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:19.547 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201317 2024-09-18T10:44:19.548 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:19.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:44:19.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201317 2024-09-18T10:44:19.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:19.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201317 2024-09-18T10:44:19.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201317' 2024-09-18T10:44:19.553 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.10 seq 236223201317 2024-09-18T10:44:19.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:44:19.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201318 -lt 236223201317 2024-09-18T10:44:19.960 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:44:19.960 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:44:19.960 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:44:20.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-18T10:44:20.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:44:20.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:44:20.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:44:20.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:44:20.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:44:20.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:44:20.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:44:20.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-18T10:44:20.952 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:44:20.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:44:20.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:44:21.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-18T10:44:21.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:44:21.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T10:44:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-09-18T10:44:21.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T10:44:21.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-09-18T10:44:21.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:44:21.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:44:21.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:44:21.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:44:21.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:44:21.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:44:21.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:21.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-09-18T10:44:21.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T10:44:21.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-09-18T10:44:21.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T10:44:21.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-09-18T10:44:21.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-09-18T10:44:25.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:44:25.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:44:25.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:44:25.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' 2024-09-18T10:44:25.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:44:25.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-09-18T10:44:25.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-09-18T10:44:25.198 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.9 2024-09-18T10:44:25.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=ce5de9db-bf3f-4574-9761-da002ab388db --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.46250/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:44:25.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-09-18T10:44:25.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-09-18T10:44:25.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:44:25.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:44:25.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:44:25.226 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:25.217+0000 7faa945a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:44:25.229 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:25.221+0000 7faa945a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:44:25.231 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:25.221+0000 7faa945a6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:44:25.535 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:25.525+0000 7faa945a6540 -1 Falling back to public interface 2024-09-18T10:44:25.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:25.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:44:26.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:44:27.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:44:27.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:27.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:44:27.049 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:44:27.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:27.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:44:27.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:44:28.337 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:44:28.325+0000 7faa945a6540 -1 osd.9 102 log_to_monitors true 2024-09-18T10:44:28.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:44:28.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:28.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:44:28.500 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:44:28.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:28.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:44:28.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:44:29.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:44:29.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:44:29.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:44:29.920 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:44:29.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:44:29.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:osd.9 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [50,102) [v2:127.0.0.1:6874/73823,v1:127.0.0.1:6875/73823] [v2:127.0.0.1:6876/73823,v1:127.0.0.1:6877/73823] exists,up 06c7b7eb-1717-4154-8d33-fdab4951ed49 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:44:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:44:30.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:44:30.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:44:30.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:44:30.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:44:30.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:44:30.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:44:30.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:44:30.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:44:30.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:44:30.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:44:30.936 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:44:31.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:44:31.537 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:44:31.537 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:44:31.538 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:44:31.538 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:44:31.538 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:44:31.538 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:44:31.538 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:44:31.538 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:44:31.539 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:44:31.539 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:44:31.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:44:31.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:31.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:44:31.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836587 2024-09-18T10:44:31.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836587 2024-09-18T10:44:31.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587' 2024-09-18T10:44:31.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:31.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:44:32.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673066 2024-09-18T10:44:32.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673066 2024-09-18T10:44:32.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066' 2024-09-18T10:44:32.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:32.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:44:32.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509536 2024-09-18T10:44:32.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509536 2024-09-18T10:44:32.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536' 2024-09-18T10:44:32.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:32.366 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:44:32.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481813 2024-09-18T10:44:32.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481813 2024-09-18T10:44:32.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813' 2024-09-18T10:44:32.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:32.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:44:32.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182486 2024-09-18T10:44:32.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182486 2024-09-18T10:44:32.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486' 2024-09-18T10:44:32.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:32.655 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:44:32.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252880 2024-09-18T10:44:32.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252880 2024-09-18T10:44:32.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486 5-360777252880' 2024-09-18T10:44:32.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:32.808 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:44:32.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729609 2024-09-18T10:44:32.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729609 2024-09-18T10:44:32.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486 5-360777252880 6-429496729609' 2024-09-18T10:44:32.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:32.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:44:33.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691903 2024-09-18T10:44:33.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691903 2024-09-18T10:44:33.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486 5-360777252880 6-429496729609 7-171798691903' 2024-09-18T10:44:33.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:33.086 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:44:33.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528378 2024-09-18T10:44:33.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528378 2024-09-18T10:44:33.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486 5-360777252880 6-429496729609 7-171798691903 8-193273528378' 2024-09-18T10:44:33.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:33.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:44:33.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500675 2024-09-18T10:44:33.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500675 2024-09-18T10:44:33.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486 5-360777252880 6-429496729609 7-171798691903 8-193273528378 9-459561500675' 2024-09-18T10:44:33.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:44:33.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:44:33.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201323 2024-09-18T10:44:33.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201323 2024-09-18T10:44:33.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836587 1-42949673066 2-64424509536 3-330712481813 4-107374182486 5-360777252880 6-429496729609 7-171798691903 8-193273528378 9-459561500675 10-236223201323' 2024-09-18T10:44:33.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:33.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836587 2024-09-18T10:44:33.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:33.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:44:33.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:33.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836587 2024-09-18T10:44:33.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836587 2024-09-18T10:44:33.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-09-18T10:44:33.554 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 21474836587 2024-09-18T10:44:33.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:44:33.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836587 -lt 21474836587 2024-09-18T10:44:33.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:33.959 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673066 2024-09-18T10:44:33.959 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:33.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:44:33.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:33.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673066 2024-09-18T10:44:33.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673066 2024-09-18T10:44:33.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-09-18T10:44:33.963 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 42949673066 2024-09-18T10:44:33.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:44:34.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673066 -lt 42949673066 2024-09-18T10:44:34.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:34.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509536 2024-09-18T10:44:34.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:34.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:44:34.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509536 2024-09-18T10:44:34.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:34.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509536 2024-09-18T10:44:34.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-18T10:44:34.448 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 64424509536 2024-09-18T10:44:34.448 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:44:34.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509536 -lt 64424509536 2024-09-18T10:44:34.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:34.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481813 2024-09-18T10:44:34.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:34.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:44:34.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:34.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481813 2024-09-18T10:44:34.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481813 2024-09-18T10:44:34.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481813' 2024-09-18T10:44:34.928 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 330712481813 2024-09-18T10:44:34.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:44:35.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481814 -lt 330712481813 2024-09-18T10:44:35.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:35.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182486 2024-09-18T10:44:35.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:35.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:44:35.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:35.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182486 2024-09-18T10:44:35.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182486 2024-09-18T10:44:35.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-09-18T10:44:35.359 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.4 seq 107374182486 2024-09-18T10:44:35.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:44:35.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182487 -lt 107374182486 2024-09-18T10:44:35.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:35.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-360777252880 2024-09-18T10:44:35.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:35.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:44:35.823 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-360777252880 2024-09-18T10:44:35.823 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:35.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252880 2024-09-18T10:44:35.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 360777252880' 2024-09-18T10:44:35.825 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.5 seq 360777252880 2024-09-18T10:44:35.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:44:36.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252880 -lt 360777252880 2024-09-18T10:44:36.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:36.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-429496729609 2024-09-18T10:44:36.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:36.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:44:36.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:36.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-429496729609 2024-09-18T10:44:36.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729609 2024-09-18T10:44:36.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 429496729609' 2024-09-18T10:44:36.234 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.6 seq 429496729609 2024-09-18T10:44:36.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:44:36.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729609 -lt 429496729609 2024-09-18T10:44:36.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:36.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:36.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691903 2024-09-18T10:44:36.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:44:36.635 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691903 2024-09-18T10:44:36.635 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:36.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691903 2024-09-18T10:44:36.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-09-18T10:44:36.637 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.7 seq 171798691903 2024-09-18T10:44:36.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:44:37.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691903 -lt 171798691903 2024-09-18T10:44:37.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:37.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528378 2024-09-18T10:44:37.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:37.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:44:37.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528378 2024-09-18T10:44:37.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:37.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528378 2024-09-18T10:44:37.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-09-18T10:44:37.048 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.8 seq 193273528378 2024-09-18T10:44:37.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:44:37.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528379 -lt 193273528378 2024-09-18T10:44:37.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:37.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-459561500675 2024-09-18T10:44:37.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:37.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:44:37.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-459561500675 2024-09-18T10:44:37.527 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:37.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500675 2024-09-18T10:44:37.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 459561500675' 2024-09-18T10:44:37.529 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.9 seq 459561500675 2024-09-18T10:44:37.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:44:37.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500676 -lt 459561500675 2024-09-18T10:44:37.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:44:37.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201323 2024-09-18T10:44:37.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:44:37.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:44:37.952 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:44:37.952 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201323 2024-09-18T10:44:37.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201323 2024-09-18T10:44:37.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201323' 2024-09-18T10:44:37.953 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.10 seq 236223201323 2024-09-18T10:44:37.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:44:38.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201324 -lt 236223201323 2024-09-18T10:44:38.351 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:44:38.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:44:38.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:44:38.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-18T10:44:38.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:44:38.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:44:38.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:44:38.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:44:38.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:44:38.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:44:38.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:44:39.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-18T10:44:39.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:44:39.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:44:39.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:44:40.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-18T10:44:40.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:44:40.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:44:40.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:325: TEST_chunk_mapping: delete_pool remap-pool 2024-09-18T10:44:40.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-09-18T10:44:40.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-09-18T10:44:40.749 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'remap-pool' does not exist 2024-09-18T10:44:40.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:326: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-09-18T10:44:41.615 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile remap-profile does not exist 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: erasure_code_plugin_exists: local plugin=isa 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: erasure_code_plugin_exists: local status 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: local grepstr 2024-09-18T10:44:41.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: erasure_code_plugin_exists: local s 2024-09-18T10:44:41.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: case `uname` in 2024-09-18T10:44:41.624 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2026: erasure_code_plugin_exists: uname 2024-09-18T10:44:41.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-09-18T10:44:41.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-09-18T10:44:42.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: erasure_code_plugin_exists: s= 2024-09-18T10:44:42.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: erasure_code_plugin_exists: local status=0 2024-09-18T10:44:42.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-18T10:44:42.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-18T10:44:43.183 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-18T10:44:43.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: erasure_code_plugin_exists: return 0 2024-09-18T10:44:43.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-09-18T10:44:43.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:194: TEST_rados_put_get_isa: local poolname=pool-isa 2024-09-18T10:44:43.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:196: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-09-18T10:44:44.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:199: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-09-18T10:44:44.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-09-18T10:44:45.078 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-isa' already exists 2024-09-18T10:44:45.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:44:46.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:202: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-09-18T10:44:46.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-18T10:44:46.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-09-18T10:44:46.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-18T10:44:46.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:46.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-18T10:44:46.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:44:47.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:44:47.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:44:47.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-18T10:44:47.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:44:47.593 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-09-18T10:44:47.593 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-09-18T10:44:47.593 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:44:47.594 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-18T10:44:47.594 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:44:48.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:10 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:9' 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-18T10:44:48.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-09-18T10:44:48.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-18T10:44:48.976 INFO:tasks.workunit.client.0.smithi174.stderr:osd.9 is already out. 2024-09-18T10:44:48.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<9\>' 2024-09-18T10:44:48.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-isa SOMETHING 2024-09-18T10:44:48.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-09-18T10:44:48.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:44:48.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-09-18T10:44:48.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:10 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T10:44:49.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-09-18T10:44:49.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:44:50.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:44:50.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 9 2024-09-18T10:44:51.954 INFO:tasks.workunit.client.0.smithi174.stderr:osd.9 is already in. 2024-09-18T10:44:51.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:44:51.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:204: TEST_rados_put_get_isa: delete_pool pool-isa 2024-09-18T10:44:51.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-09-18T10:44:51.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-09-18T10:44:53.663 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-isa' does not exist 2024-09-18T10:44:53.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:44:53.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:210: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:44:53.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-18T10:44:53.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:44:54.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:44:55.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:44:55.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-18T10:44:55.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:44:55.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-09-18T10:44:55.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T10:44:55.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:44:55.381 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T10:44:55.381 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:44:56.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr:9' 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-09-18T10:44:56.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-09-18T10:44:57.072 INFO:tasks.workunit.client.0.smithi174.stderr:osd.9 is already out. 2024-09-18T10:44:57.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds ecpool SOMETHING 2024-09-18T10:44:57.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-09-18T10:44:57.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:44:57.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<9\>' 2024-09-18T10:44:57.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-18T10:44:57.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:44:57.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-09-18T10:44:57.913 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:44:57.913 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:44:57.913 INFO:tasks.workunit.client.0.smithi174.stderr:5' 2024-09-18T10:44:57.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-09-18T10:44:57.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:44:59.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:44:59.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 9 2024-09-18T10:45:00.567 INFO:tasks.workunit.client.0.smithi174.stderr:osd.9 is already in. 2024-09-18T10:45:00.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:45:00.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-09-18T10:45:00.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:213: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-09-18T10:45:00.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:215: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-09-18T10:45:02.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-09-18T10:45:02.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-09-18T10:45:03.763 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T10:45:03.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:45:04.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:222: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-18T10:45:04.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:45:06.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:45:06.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:45:06.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-18T10:45:06.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:45:06.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-18T10:45:06.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:45:06.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:06.492 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-18T10:45:06.492 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:07.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-18T10:45:07.330 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:07.330 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:07.330 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:07.330 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:07.330 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:07.331 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-18T10:45:07.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-18T10:45:07.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-09-18T10:45:07.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-09-18T10:45:08.140 INFO:tasks.workunit.client.0.smithi174.stderr:osd.10 is already out. 2024-09-18T10:45:08.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-jerasure SOMETHING 2024-09-18T10:45:08.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:45:08.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:08.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<10\>' 2024-09-18T10:45:08.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-18T10:45:08.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-09-18T10:45:08.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:45:09.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:45:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 10 2024-09-18T10:45:12.037 INFO:tasks.workunit.client.0.smithi174.stderr:osd.10 is already in. 2024-09-18T10:45:12.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_osds_out_in: local dir=td/test-erasure-code 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:101: rados_osds_out_in: local poolname=pool-jerasure 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:102: rados_osds_out_in: local objname=SOMETHING 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-09-18T10:45:12.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-09-18T10:45:12.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: printf '%*s' 1024 IIII 2024-09-18T10:45:12.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:112: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:45:14.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:113: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:45:14.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:114: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:45:14.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:115: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-09-18T10:45:14.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:123: rados_osds_out_in: wait_for_clean 2024-09-18T10:45:14.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:45:14.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:45:14.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:45:14.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:45:14.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:45:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:45:14.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:45:14.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:45:14.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:45:14.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:45:14.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:45:14.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:45:14.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:45:14.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:45:14.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:45:15.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:45:15.467 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:15.467 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:45:15.467 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:15.467 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:15.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:45:15.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836598 2024-09-18T10:45:15.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836598 2024-09-18T10:45:15.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598' 2024-09-18T10:45:15.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:15.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:45:16.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673077 2024-09-18T10:45:16.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673077 2024-09-18T10:45:16.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077' 2024-09-18T10:45:16.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:16.289 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:45:16.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509547 2024-09-18T10:45:16.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509547 2024-09-18T10:45:16.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547' 2024-09-18T10:45:16.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:16.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:45:16.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481824 2024-09-18T10:45:16.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481824 2024-09-18T10:45:16.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824' 2024-09-18T10:45:16.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:16.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:45:17.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182497 2024-09-18T10:45:17.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182497 2024-09-18T10:45:17.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497' 2024-09-18T10:45:17.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:17.140 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:45:17.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252891 2024-09-18T10:45:17.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252891 2024-09-18T10:45:17.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497 5-360777252891' 2024-09-18T10:45:17.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:17.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:45:17.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729620 2024-09-18T10:45:17.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729620 2024-09-18T10:45:17.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497 5-360777252891 6-429496729620' 2024-09-18T10:45:17.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:17.485 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:45:17.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691914 2024-09-18T10:45:17.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691914 2024-09-18T10:45:17.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497 5-360777252891 6-429496729620 7-171798691914' 2024-09-18T10:45:17.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:17.656 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:45:17.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528389 2024-09-18T10:45:17.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528389 2024-09-18T10:45:17.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497 5-360777252891 6-429496729620 7-171798691914 8-193273528389' 2024-09-18T10:45:17.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:17.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:45:18.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500686 2024-09-18T10:45:18.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500686 2024-09-18T10:45:18.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497 5-360777252891 6-429496729620 7-171798691914 8-193273528389 9-459561500686' 2024-09-18T10:45:18.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:18.139 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:45:18.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201334 2024-09-18T10:45:18.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201334 2024-09-18T10:45:18.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836598 1-42949673077 2-64424509547 3-330712481824 4-107374182497 5-360777252891 6-429496729620 7-171798691914 8-193273528389 9-459561500686 10-236223201334' 2024-09-18T10:45:18.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:18.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836598 2024-09-18T10:45:18.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:18.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:45:18.365 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836598 2024-09-18T10:45:18.365 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:18.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836598 2024-09-18T10:45:18.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-09-18T10:45:18.367 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836598 2024-09-18T10:45:18.368 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:45:18.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836598 -lt 21474836598 2024-09-18T10:45:18.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:18.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673077 2024-09-18T10:45:18.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:18.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:45:18.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673077 2024-09-18T10:45:18.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:18.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673077 2024-09-18T10:45:18.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-09-18T10:45:18.982 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949673077 2024-09-18T10:45:18.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:45:19.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673077 -lt 42949673077 2024-09-18T10:45:19.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:19.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509547 2024-09-18T10:45:19.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:19.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:45:19.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509547 2024-09-18T10:45:19.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:19.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509547 2024-09-18T10:45:19.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509547' 2024-09-18T10:45:19.399 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509547 2024-09-18T10:45:19.399 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:45:19.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509548 -lt 64424509547 2024-09-18T10:45:19.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:19.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481824 2024-09-18T10:45:19.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:19.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:45:19.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481824 2024-09-18T10:45:19.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:19.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481824 2024-09-18T10:45:19.845 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 330712481824 2024-09-18T10:45:19.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481824' 2024-09-18T10:45:19.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:45:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481825 -lt 330712481824 2024-09-18T10:45:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:20.348 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182497 2024-09-18T10:45:20.348 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:20.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:45:20.351 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:20.351 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182497 2024-09-18T10:45:20.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182497 2024-09-18T10:45:20.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182497' 2024-09-18T10:45:20.352 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 107374182497 2024-09-18T10:45:20.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:45:20.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182497 -lt 107374182497 2024-09-18T10:45:20.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:20.781 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-360777252891 2024-09-18T10:45:20.782 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:20.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:45:20.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-360777252891 2024-09-18T10:45:20.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:20.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252891 2024-09-18T10:45:20.788 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 360777252891 2024-09-18T10:45:20.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 360777252891' 2024-09-18T10:45:20.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:45:21.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252891 -lt 360777252891 2024-09-18T10:45:21.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:21.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-429496729620 2024-09-18T10:45:21.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:21.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:45:21.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-429496729620 2024-09-18T10:45:21.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:21.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729620 2024-09-18T10:45:21.187 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 429496729620 2024-09-18T10:45:21.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 429496729620' 2024-09-18T10:45:21.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:45:21.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729621 -lt 429496729620 2024-09-18T10:45:21.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:21.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691914 2024-09-18T10:45:21.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:21.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:45:21.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691914 2024-09-18T10:45:21.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:21.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691914 2024-09-18T10:45:21.673 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.7 seq 171798691914 2024-09-18T10:45:21.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691914' 2024-09-18T10:45:21.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:45:22.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691915 -lt 171798691914 2024-09-18T10:45:22.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:22.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528389 2024-09-18T10:45:22.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:22.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:45:22.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528389 2024-09-18T10:45:22.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:22.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528389 2024-09-18T10:45:22.176 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 193273528389 2024-09-18T10:45:22.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528389' 2024-09-18T10:45:22.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:45:22.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528390 -lt 193273528389 2024-09-18T10:45:22.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:22.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-459561500686 2024-09-18T10:45:22.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:22.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:45:22.624 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-459561500686 2024-09-18T10:45:22.624 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:22.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500686 2024-09-18T10:45:22.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 459561500686' 2024-09-18T10:45:22.627 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.9 seq 459561500686 2024-09-18T10:45:22.627 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:45:23.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500687 -lt 459561500686 2024-09-18T10:45:23.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:23.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201334 2024-09-18T10:45:23.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:23.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:45:23.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201334 2024-09-18T10:45:23.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:23.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201334 2024-09-18T10:45:23.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201334' 2024-09-18T10:45:23.078 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.10 seq 236223201334 2024-09-18T10:45:23.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:45:23.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201334 -lt 236223201334 2024-09-18T10:45:23.483 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:45:23.484 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:45:23.484 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:45:24.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-18T10:45:24.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:45:24.003 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:45:24.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:45:24.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:45:24.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:45:24.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:45:24.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:45:24.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-18T10:45:24.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:45:24.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:45:24.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:45:24.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-18T10:45:24.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:45:24.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:45:24.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:124: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-18T10:45:24.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:45:24.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:24.909 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-18T10:45:24.909 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:25.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-18T10:45:25.309 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:25.309 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:25.309 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:25.309 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:124: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:125: rados_osds_out_in: osds=($osds_list) 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:125: rados_osds_out_in: local -a osds 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:126: rados_osds_out_in: for osd in 0 1 2024-09-18T10:45:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: ceph osd out 5 2024-09-18T10:45:25.979 INFO:tasks.workunit.client.0.smithi174.stderr:osd.5 is already out. 2024-09-18T10:45:25.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:126: rados_osds_out_in: for osd in 0 1 2024-09-18T10:45:25.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: ceph osd out 1 2024-09-18T10:45:27.467 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already out. 2024-09-18T10:45:27.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: wait_for_clean 2024-09-18T10:45:27.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:45:27.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:45:27.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:45:27.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:45:27.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:45:27.477 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:45:27.477 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:45:27.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:45:27.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:45:27.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:45:27.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:45:27.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:45:27.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:45:27.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:45:27.656 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:45:28.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:45:28.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:28.481 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:45:28.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836602 2024-09-18T10:45:28.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836602 2024-09-18T10:45:28.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602' 2024-09-18T10:45:28.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:28.926 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:45:29.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673082 2024-09-18T10:45:29.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673082 2024-09-18T10:45:29.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082' 2024-09-18T10:45:29.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:29.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:45:29.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509552 2024-09-18T10:45:29.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509552 2024-09-18T10:45:29.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552' 2024-09-18T10:45:29.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:29.596 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:45:29.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481829 2024-09-18T10:45:29.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481829 2024-09-18T10:45:29.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829' 2024-09-18T10:45:29.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:29.747 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:45:30.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182501 2024-09-18T10:45:30.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182501 2024-09-18T10:45:30.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501' 2024-09-18T10:45:30.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:30.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:45:30.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252895 2024-09-18T10:45:30.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252895 2024-09-18T10:45:30.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501 5-360777252895' 2024-09-18T10:45:30.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:30.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:45:30.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729625 2024-09-18T10:45:30.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729625 2024-09-18T10:45:30.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501 5-360777252895 6-429496729625' 2024-09-18T10:45:30.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:30.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:45:30.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691918 2024-09-18T10:45:30.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691918 2024-09-18T10:45:30.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501 5-360777252895 6-429496729625 7-171798691918' 2024-09-18T10:45:30.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:30.988 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:45:31.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528394 2024-09-18T10:45:31.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528394 2024-09-18T10:45:31.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501 5-360777252895 6-429496729625 7-171798691918 8-193273528394' 2024-09-18T10:45:31.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:31.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:45:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500691 2024-09-18T10:45:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500691 2024-09-18T10:45:31.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501 5-360777252895 6-429496729625 7-171798691918 8-193273528394 9-459561500691' 2024-09-18T10:45:31.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:31.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:45:31.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201338 2024-09-18T10:45:31.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201338 2024-09-18T10:45:31.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836602 1-42949673082 2-64424509552 3-330712481829 4-107374182501 5-360777252895 6-429496729625 7-171798691918 8-193273528394 9-459561500691 10-236223201338' 2024-09-18T10:45:31.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:31.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836602 2024-09-18T10:45:31.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:31.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:45:31.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836602 2024-09-18T10:45:31.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:31.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836602 2024-09-18T10:45:31.466 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836602 2024-09-18T10:45:31.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2024-09-18T10:45:31.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:45:32.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836603 -lt 21474836602 2024-09-18T10:45:32.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:32.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673082 2024-09-18T10:45:32.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:32.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:45:32.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:32.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673082 2024-09-18T10:45:32.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673082 2024-09-18T10:45:32.120 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949673082 2024-09-18T10:45:32.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673082' 2024-09-18T10:45:32.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:45:32.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673082 -lt 42949673082 2024-09-18T10:45:32.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:32.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509552 2024-09-18T10:45:32.626 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:32.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:45:32.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509552 2024-09-18T10:45:32.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:32.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509552 2024-09-18T10:45:32.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509552' 2024-09-18T10:45:32.632 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509552 2024-09-18T10:45:32.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:45:33.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509552 -lt 64424509552 2024-09-18T10:45:33.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:33.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481829 2024-09-18T10:45:33.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:33.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:45:33.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481829 2024-09-18T10:45:33.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:33.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481829 2024-09-18T10:45:33.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481829' 2024-09-18T10:45:33.126 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 330712481829 2024-09-18T10:45:33.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:45:33.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481830 -lt 330712481829 2024-09-18T10:45:33.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:33.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182501 2024-09-18T10:45:33.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:33.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:45:33.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182501 2024-09-18T10:45:33.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:33.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182501 2024-09-18T10:45:33.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182501' 2024-09-18T10:45:33.584 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 107374182501 2024-09-18T10:45:33.584 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:45:34.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182502 -lt 107374182501 2024-09-18T10:45:34.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:34.028 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-360777252895 2024-09-18T10:45:34.028 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:34.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:45:34.031 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-360777252895 2024-09-18T10:45:34.031 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:34.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252895 2024-09-18T10:45:34.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 360777252895' 2024-09-18T10:45:34.033 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 360777252895 2024-09-18T10:45:34.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:45:34.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252896 -lt 360777252895 2024-09-18T10:45:34.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:34.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-429496729625 2024-09-18T10:45:34.478 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:34.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:45:34.481 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-429496729625 2024-09-18T10:45:34.481 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:34.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729625 2024-09-18T10:45:34.484 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 429496729625 2024-09-18T10:45:34.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 429496729625' 2024-09-18T10:45:34.485 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:45:34.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729625 -lt 429496729625 2024-09-18T10:45:34.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:34.921 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:34.921 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691918 2024-09-18T10:45:34.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:45:34.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691918 2024-09-18T10:45:34.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:34.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691918 2024-09-18T10:45:34.926 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.7 seq 171798691918 2024-09-18T10:45:34.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691918' 2024-09-18T10:45:34.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:45:35.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691919 -lt 171798691918 2024-09-18T10:45:35.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:35.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528394 2024-09-18T10:45:35.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:35.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:45:35.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528394 2024-09-18T10:45:35.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:35.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528394 2024-09-18T10:45:35.344 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 193273528394 2024-09-18T10:45:35.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528394' 2024-09-18T10:45:35.345 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:45:35.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528394 -lt 193273528394 2024-09-18T10:45:35.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:35.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:35.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-459561500691 2024-09-18T10:45:35.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:45:35.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-459561500691 2024-09-18T10:45:35.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:35.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500691 2024-09-18T10:45:35.847 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.9 seq 459561500691 2024-09-18T10:45:35.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 459561500691' 2024-09-18T10:45:35.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:45:36.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500692 -lt 459561500691 2024-09-18T10:45:36.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:36.289 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201338 2024-09-18T10:45:36.289 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:36.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:45:36.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201338 2024-09-18T10:45:36.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:36.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201338 2024-09-18T10:45:36.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201338' 2024-09-18T10:45:36.293 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.10 seq 236223201338 2024-09-18T10:45:36.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:45:36.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201339 -lt 236223201338 2024-09-18T10:45:36.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:45:36.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:45:36.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:45:37.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-18T10:45:37.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:45:37.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:45:37.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:45:37.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:45:37.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:45:37.231 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:45:37.231 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:45:37.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-18T10:45:37.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:45:37.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:45:37.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:45:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-18T10:45:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:45:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:45:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: for osd in 0 1 2024-09-18T10:45:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-18T10:45:38.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:45:38.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:38.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: grep '\<5\>' 2024-09-18T10:45:38.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-18T10:45:38.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:38.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T10:45:38.651 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:45:38.651 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:38.651 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:38.651 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:38.651 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:38.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-09-18T10:45:38.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: for osd in 0 1 2024-09-18T10:45:38.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-18T10:45:38.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:45:38.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:38.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:134: rados_osds_out_in: grep '\<1\>' 2024-09-18T10:45:38.653 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:38.653 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-18T10:45:39.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-09-18T10:45:39.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:136: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:45:39.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:45:39.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:143: rados_osds_out_in: for osd in 0 1 2024-09-18T10:45:39.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:144: rados_osds_out_in: ceph osd in 5 2024-09-18T10:45:39.799 INFO:tasks.workunit.client.0.smithi174.stderr:osd.5 is already in. 2024-09-18T10:45:39.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:143: rados_osds_out_in: for osd in 0 1 2024-09-18T10:45:39.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:144: rados_osds_out_in: ceph osd in 1 2024-09-18T10:45:41.573 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already in. 2024-09-18T10:45:41.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:146: rados_osds_out_in: wait_for_clean 2024-09-18T10:45:41.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:45:41.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:45:41.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:45:41.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:45:41.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:45:41.583 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:45:41.583 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:45:41.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:45:41.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:45:41.584 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:45:41.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:45:41.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:45:41.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:45:41.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:45:41.778 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:45:42.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:45:42.381 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:42.381 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:8 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:42.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:45:42.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836607 2024-09-18T10:45:42.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836607 2024-09-18T10:45:42.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607' 2024-09-18T10:45:42.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:42.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:45:43.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673086 2024-09-18T10:45:43.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673086 2024-09-18T10:45:43.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086' 2024-09-18T10:45:43.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:43.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:45:43.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509556 2024-09-18T10:45:43.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509556 2024-09-18T10:45:43.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556' 2024-09-18T10:45:43.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:43.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:45:43.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481834 2024-09-18T10:45:43.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481834 2024-09-18T10:45:43.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834' 2024-09-18T10:45:43.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:43.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:45:44.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182506 2024-09-18T10:45:44.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182506 2024-09-18T10:45:44.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506' 2024-09-18T10:45:44.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:44.118 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:45:44.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252900 2024-09-18T10:45:44.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252900 2024-09-18T10:45:44.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506 5-360777252900' 2024-09-18T10:45:44.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:44.390 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:45:44.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=429496729629 2024-09-18T10:45:44.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 429496729629 2024-09-18T10:45:44.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506 5-360777252900 6-429496729629' 2024-09-18T10:45:44.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:44.801 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T10:45:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691923 2024-09-18T10:45:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691923 2024-09-18T10:45:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506 5-360777252900 6-429496729629 7-171798691923' 2024-09-18T10:45:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:44.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T10:45:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528398 2024-09-18T10:45:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528398 2024-09-18T10:45:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506 5-360777252900 6-429496729629 7-171798691923 8-193273528398' 2024-09-18T10:45:45.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:45.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-18T10:45:45.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500696 2024-09-18T10:45:45.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500696 2024-09-18T10:45:45.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506 5-360777252900 6-429496729629 7-171798691923 8-193273528398 9-459561500696' 2024-09-18T10:45:45.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:45:45.441 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-09-18T10:45:45.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=236223201343 2024-09-18T10:45:45.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 236223201343 2024-09-18T10:45:45.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836607 1-42949673086 2-64424509556 3-330712481834 4-107374182506 5-360777252900 6-429496729629 7-171798691923 8-193273528398 9-459561500696 10-236223201343' 2024-09-18T10:45:45.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:45.661 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836607 2024-09-18T10:45:45.661 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:45.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:45:45.664 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836607 2024-09-18T10:45:45.664 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:45.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836607 2024-09-18T10:45:45.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-09-18T10:45:45.666 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 21474836607 2024-09-18T10:45:45.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:45:46.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836608 -lt 21474836607 2024-09-18T10:45:46.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:46.210 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673086 2024-09-18T10:45:46.210 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:46.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:45:46.214 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673086 2024-09-18T10:45:46.214 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:46.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673086 2024-09-18T10:45:46.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673086' 2024-09-18T10:45:46.217 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 42949673086 2024-09-18T10:45:46.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:45:46.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673087 -lt 42949673086 2024-09-18T10:45:46.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:46.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509556 2024-09-18T10:45:46.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:46.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:45:46.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509556 2024-09-18T10:45:46.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:46.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509556 2024-09-18T10:45:46.751 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 64424509556 2024-09-18T10:45:46.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2024-09-18T10:45:46.752 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:45:47.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509557 -lt 64424509556 2024-09-18T10:45:47.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:47.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-330712481834 2024-09-18T10:45:47.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:47.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:45:47.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-330712481834 2024-09-18T10:45:47.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:47.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481834 2024-09-18T10:45:47.206 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 330712481834 2024-09-18T10:45:47.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 330712481834' 2024-09-18T10:45:47.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:45:47.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481834 -lt 330712481834 2024-09-18T10:45:47.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:47.683 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182506 2024-09-18T10:45:47.684 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:47.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:45:47.687 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182506 2024-09-18T10:45:47.687 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:47.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182506 2024-09-18T10:45:47.689 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 107374182506 2024-09-18T10:45:47.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182506' 2024-09-18T10:45:47.690 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:45:48.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182507 -lt 107374182506 2024-09-18T10:45:48.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:48.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-360777252900 2024-09-18T10:45:48.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:48.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:45:48.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-360777252900 2024-09-18T10:45:48.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:48.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252900 2024-09-18T10:45:48.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 360777252900' 2024-09-18T10:45:48.176 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 360777252900 2024-09-18T10:45:48.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:45:48.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252901 -lt 360777252900 2024-09-18T10:45:48.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:48.643 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-429496729629 2024-09-18T10:45:48.643 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:48.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:45:48.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:48.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-429496729629 2024-09-18T10:45:48.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=429496729629 2024-09-18T10:45:48.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 429496729629' 2024-09-18T10:45:48.649 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 429496729629 2024-09-18T10:45:48.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:45:49.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 429496729630 -lt 429496729629 2024-09-18T10:45:49.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:49.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-171798691923 2024-09-18T10:45:49.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:49.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T10:45:49.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-171798691923 2024-09-18T10:45:49.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:49.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691923 2024-09-18T10:45:49.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 171798691923' 2024-09-18T10:45:49.140 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.7 seq 171798691923 2024-09-18T10:45:49.141 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T10:45:49.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691924 -lt 171798691923 2024-09-18T10:45:49.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:49.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-193273528398 2024-09-18T10:45:49.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:49.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T10:45:49.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-193273528398 2024-09-18T10:45:49.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:49.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528398 2024-09-18T10:45:49.554 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 193273528398 2024-09-18T10:45:49.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 193273528398' 2024-09-18T10:45:49.555 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T10:45:49.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528399 -lt 193273528398 2024-09-18T10:45:49.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:49.951 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:49.951 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 9-459561500696 2024-09-18T10:45:49.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=9 2024-09-18T10:45:49.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 9-459561500696 2024-09-18T10:45:49.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:49.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500696 2024-09-18T10:45:49.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.9 seq 459561500696' 2024-09-18T10:45:49.956 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.9 seq 459561500696 2024-09-18T10:45:49.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-18T10:45:50.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500697 -lt 459561500696 2024-09-18T10:45:50.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:45:50.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:45:50.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 10-236223201343 2024-09-18T10:45:50.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=10 2024-09-18T10:45:50.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 10-236223201343 2024-09-18T10:45:50.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:45:50.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=236223201343 2024-09-18T10:45:50.418 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.10 seq 236223201343 2024-09-18T10:45:50.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.10 seq 236223201343' 2024-09-18T10:45:50.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 10 2024-09-18T10:45:50.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 236223201344 -lt 236223201343 2024-09-18T10:45:50.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:45:50.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:45:50.873 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:45:51.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-09-18T10:45:51.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:45:51.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:45:51.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:45:51.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:45:51.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:45:51.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:45:51.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:45:51.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-09-18T10:45:51.856 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:45:51.856 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:45:51.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:45:52.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-09-18T10:45:52.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:45:52.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:45:52.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-09-18T10:45:52.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:45:52.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-09-18T10:45:52.371 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:52.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-18T10:45:52.854 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:45:52.854 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:52.854 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:45:52.854 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:45:52.854 INFO:tasks.workunit.client.0.smithi174.stderr:10' 2024-09-18T10:45:52.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-09-18T10:45:52.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-09-18T10:45:52.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:45:52.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:225: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-09-18T10:45:52.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-09-18T10:45:52.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T10:45:53.441 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T10:45:53.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-09-18T10:45:54.534 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile profile-jerasure does not exist 2024-09-18T10:45:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:45:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-09-18T10:45:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-09-18T10:45:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:153: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-09-18T10:45:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:154: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-09-18T10:45:54.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-09-18T10:45:55.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:161: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-18T10:45:55.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-09-18T10:45:56.536 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-lrc-a' already exists 2024-09-18T10:45:56.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:164: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-18T10:45:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-18T10:45:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-18T10:45:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:45:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-18T10:45:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:45:58.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:45:59.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:45:59.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-18T10:45:59.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:45:59.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-18T10:45:59.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-09-18T10:45:59.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:45:59.128 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-18T10:45:59.128 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:45:59.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-18T10:45:59.765 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:45:59.765 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T10:45:59.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-09-18T10:45:59.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-18T10:45:59.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-18T10:45:59.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-09-18T10:46:00.454 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 is already out. 2024-09-18T10:46:00.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-09-18T10:46:00.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-09-18T10:46:00.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<3\>' 2024-09-18T10:46:00.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:46:00.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-09-18T10:46:00.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:46:01.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-09-18T10:46:01.204 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:46:01.204 INFO:tasks.workunit.client.0.smithi174.stderr:7' 2024-09-18T10:46:01.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-09-18T10:46:01.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:46:01.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:46:01.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 3 2024-09-18T10:46:03.277 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 is already in. 2024-09-18T10:46:03.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:46:03.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:166: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-09-18T10:46:03.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-09-18T10:46:03.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-09-18T10:46:05.427 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-lrc-a' does not exist 2024-09-18T10:46:05.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:167: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-09-18T10:46:06.951 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile profile-lrc-a does not exist 2024-09-18T10:46:06.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:46:06.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-09-18T10:46:06.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-09-18T10:46:06.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:172: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-09-18T10:46:06.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:173: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-09-18T10:46:06.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-09-18T10:46:08.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-09-18T10:46:08.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-09-18T10:46:09.132 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-lrc' already exists 2024-09-18T10:46:09.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:182: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-18T10:46:10.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:10.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-18T10:46:10.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:10.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-18T10:46:10.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:46:11.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:46:12.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:46:12.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-18T10:46:12.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:46:12.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-18T10:46:12.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-09-18T10:46:12.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:46:12.101 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-18T10:46:12.101 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:46:12.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-09-18T10:46:12.607 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:46:12.607 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr:10 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr:7' 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-09-18T10:46:12.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-18T10:46:12.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-09-18T10:46:12.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-09-18T10:46:13.260 INFO:tasks.workunit.client.0.smithi174.stderr:osd.7 is already out. 2024-09-18T10:46:13.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-lrc SOMETHING 2024-09-18T10:46:13.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-09-18T10:46:13.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:46:13.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<7\>' 2024-09-18T10:46:13.273 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:46:13.273 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-09-18T10:46:13.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:10 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:4' 2024-09-18T10:46:13.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-09-18T10:46:13.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:46:17.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:46:17.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 7 2024-09-18T10:46:18.932 INFO:tasks.workunit.client.0.smithi174.stderr:osd.7 is already in. 2024-09-18T10:46:18.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:46:18.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:184: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-09-18T10:46:18.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-09-18T10:46:18.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-09-18T10:46:20.748 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-lrc' does not exist 2024-09-18T10:46:20.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:185: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-09-18T10:46:22.154 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile profile-lrc does not exist 2024-09-18T10:46:22.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-09-18T10:46:22.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-09-18T10:46:22.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-09-18T10:46:22.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:232: TEST_rados_put_get_shec: local poolname=pool-shec 2024-09-18T10:46:22.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:233: TEST_rados_put_get_shec: local profile=profile-shec 2024-09-18T10:46:22.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:235: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-09-18T10:46:23.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-09-18T10:46:23.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-09-18T10:46:24.257 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-shec' already exists 2024-09-18T10:46:24.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:46:25.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:242: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-09-18T10:46:25.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-09-18T10:46:25.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-09-18T10:46:25.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-09-18T10:46:25.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-09-18T10:46:25.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-09-18T10:46:25.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-09-18T10:46:26.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:46:27.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:46:27.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-09-18T10:46:27.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:46:27.158 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-09-18T10:46:27.158 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-09-18T10:46:27.158 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:46:27.159 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-18T10:46:27.159 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:46:27.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-09-18T10:46:27.581 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:46:27.582 INFO:tasks.workunit.client.0.smithi174.stderr:1' 2024-09-18T10:46:27.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-09-18T10:46:27.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-09-18T10:46:27.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-09-18T10:46:27.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-09-18T10:46:28.272 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already out. 2024-09-18T10:46:28.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: get_osds pool-shec SOMETHING 2024-09-18T10:46:28.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-09-18T10:46:28.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:91: rados_put_get: grep '\<1\>' 2024-09-18T10:46:28.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-09-18T10:46:28.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-09-18T10:46:28.287 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:46:29.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-09-18T10:46:29.227 INFO:tasks.workunit.client.0.smithi174.stderr:9 2024-09-18T10:46:29.227 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T10:46:29.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-09-18T10:46:29.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:92: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-09-18T10:46:30.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-09-18T10:46:30.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: ceph osd in 1 2024-09-18T10:46:31.349 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 is already in. 2024-09-18T10:46:31.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-09-18T10:46:31.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:244: TEST_rados_put_get_shec: delete_pool pool-shec 2024-09-18T10:46:31.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-09-18T10:46:31.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-09-18T10:46:33.226 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-shec' does not exist 2024-09-18T10:46:33.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:245: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-09-18T10:46:35.083 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile profile-shec does not exist 2024-09-18T10:46:35.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-09-18T10:46:35.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-09-18T10:46:35.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-09-18T10:46:35.994 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'ecpool' does not exist 2024-09-18T10:46:36.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-09-18T10:46:36.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-18T10:46:36.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:46:36.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-18T10:46:36.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:46:36.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:46:36.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:46:36.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:46:36.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:46:36.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:46:36.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:46:36.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:46:36.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:46:36.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:46:36.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:46:36.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:46:36.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:46:36.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:46:36.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:46:36.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:46:36.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:46:36.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:46:36.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-18T10:46:36.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:46:36.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:36.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:46:36.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46250 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-erasure-code 0 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-09-18T10:46:36.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:46:36.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:46:36.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:46:36.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:46:36.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:46:36.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:46:36.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:46:36.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:46:36.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:46:36.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:46:36.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:46:36.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:46:36.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:46:36.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:46:36.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:46:36.247 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:46:36.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:46:36.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T10:46:36.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-09-18T10:46:36.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:46:36.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:36.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46250 2024-09-18T10:46:36.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46250 2024-09-18T10:46:36.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:46:36.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:46:36.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T10:46:36.255 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T10:46:36.256 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T10:46:36.267 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-09-18T10:46:36.267 DEBUG:teuthology.orchestra.run.smithi174:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7061ea6ca50908d104e9f3979dc615e118d9e153 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-09-18T10:46:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-18T10:46:36.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/test-erasure-eio 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-09-18T10:46:36.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-09-18T10:46:36.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-09-18T10:46:36.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none ' 2024-09-18T10:46:36.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-09-18T10:46:36.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-18T10:46:36.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-09-18T10:46:36.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-18T10:46:36.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_ec_object_attr_read_error 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_ec_recovery_multiple_errors 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_ec_recovery_multiple_objects 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_ec_recovery_multiple_objects_eio 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_ec_recovery_unfound 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_ec_single_recovery_error 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_bad_size_shard_0 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_bad_size_shard_1 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_subread_eio_shard_0 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_subread_eio_shard_1 2024-09-18T10:46:36.307 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_subread_missing 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:46:36.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:46:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:46:36.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:46:36.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:46:36.310 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:46:36.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:46:36.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:46:36.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:46:36.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:46:36.314 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:46:36.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:46:36.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:46:36.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:46:36.316 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:46:36.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:46:36.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:46:36.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T10:46:36.321 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:46:36.321 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:36.321 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:36.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T10:46:36.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:46:36.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:46:36.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T10:46:36.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:46:36.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:36.325 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:36.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T10:46:36.327 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:46:36.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:46:36.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:46:36.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T10:46:36.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T10:46:36.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:46:36.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:46:36.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:46:36.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:46:36.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:36.732 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:36.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:46:36.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T10:46:37.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:37.050 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:46:37.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:46:37.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T10:46:37.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:46:37.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:46:37.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:46:37.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:46:37.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:46:37.115 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:46:37.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T10:46:37.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T10:46:37.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T10:46:37.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:46:37.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:46:37.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:46:37.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T10:46:37.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:46:38.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:46:38.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:46:38.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:46:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:46:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:38.133 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:38.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:46:38.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:46:38.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:46:38.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T10:46:38.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T10:46:38.494 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T10:46:38.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:46:39.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T10:46:39.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:46:39.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:46:39.506 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:46:39.506 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:39.506 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:39.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:46:39.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T10:46:39.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T10:46:39.583 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:46:39.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T10:46:39.594 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:46:36.767+0000 7f2055edea00 0 load: jerasure load: lrc 2024-09-18T10:46:39.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-18T10:46:39.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T10:46:39.597 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-18T10:46:39.597 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-18T10:46:39.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:46:39.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T10:46:39.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:46:39.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:46:39.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:46:39.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:46:39.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:46:39.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:46:39.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:46:39.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:46:39.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:46:39.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:46:39.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:46:39.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:46:39.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:46:39.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T10:46:39.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:46:39.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcd88305-2757-4169-99a9-cc6f256a8524 2024-09-18T10:46:39.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dcd88305-2757-4169-99a9-cc6f256a8524' 2024-09-18T10:46:39.605 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 dcd88305-2757-4169-99a9-cc6f256a8524 2024-09-18T10:46:39.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:46:39.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPr+pmLKQjJRAAoREp59hfK9jofoFyOgI/ew== 2024-09-18T10:46:39.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPr+pmLKQjJRAAoREp59hfK9jofoFyOgI/ew=="}' 2024-09-18T10:46:39.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcd88305-2757-4169-99a9-cc6f256a8524 -i td/test-erasure-eio/0/new.json 2024-09-18T10:46:39.982 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:46:39.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T10:46:39.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPr+pmLKQjJRAAoREp59hfK9jofoFyOgI/ew== --osd-uuid dcd88305-2757-4169-99a9-cc6f256a8524 2024-09-18T10:46:40.029 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.023+0000 7f2e54dd7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:46:40.032 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.023+0000 7f2e54dd7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:46:40.033 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.027+0000 7f2e54dd7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:46:40.034 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.027+0000 7f2e54dd7540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:46:40.034 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.027+0000 7f2e54dd7540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:46:40.035 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.027+0000 7f2e54dd7540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:46:40.122 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:40.115+0000 7f2e54dd7540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T10:46:45.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T10:46:45.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:46:45.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:46:45.576 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:46:45.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:46:46.125 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:46:46.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:46:46.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:46:46.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:46:46.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:46:46.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:46:46.156 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:46.147+0000 7f44b104a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:46:46.158 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:46.151+0000 7f44b104a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:46:46.160 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:46.151+0000 7f44b104a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:46:46.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:46:46.523 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:46:46.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:46.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:46.711 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:46.703+0000 7f44b104a540 -1 Falling back to public interface 2024-09-18T10:46:46.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:47.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:47.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:47.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:46:47.962 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:46:47.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:47.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:48.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:49.370 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:46:49.363+0000 7f44b104a540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:46:49.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:49.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:49.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:46:49.384 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:46:49.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:49.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:49.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:50.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:50.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:50.807 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:46:50.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:46:50.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:50.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:51.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:52.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:52.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:52.289 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:46:52.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:46:52.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:52.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:52.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:53.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:53.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:53.711 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:46:53.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:46:53.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:53.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:54.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:55.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:55.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:55.228 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:46:55.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:46:55.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:55.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:55.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:56.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:56.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:56.675 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:46:56.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:46:56.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:56.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:57.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:58.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:58.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:58.153 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:46:58.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:46:58.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:58.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:46:58.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:46:59.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:46:59.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:46:59.619 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:46:59.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:46:59.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:46:59.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:00.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:01.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:01.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:01.112 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:47:01.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:47:01.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:01.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:01.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:02.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:02.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:02.644 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:47:02.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:47:02.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:02.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:03.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:04.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:04.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:04.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:47:04.070 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:47:04.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:04.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:04.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:05.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:05.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:05.971 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:47:05.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:47:05.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:05.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:06.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:07.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:07.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:07.555 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:47:07.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:47:07.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:07.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:08.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:09.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:09.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:09.474 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:47:09.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:47:09.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:09.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:10.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:11.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:11.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:11.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:47:11.233 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:47:11.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:11.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:11.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:12.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:12.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:12.644 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:47:12.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:47:12.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:12.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:47:13.066 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/80709,v1:127.0.0.1:6803/80709] [v2:127.0.0.1:6804/80709,v1:127.0.0.1:6805/80709] exists,up dcd88305-2757-4169-99a9-cc6f256a8524 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T10:47:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:47:13.070 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:47:13.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:47:13.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:47:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:47:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:47:13.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:47:13.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T10:47:13.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:47:13.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a21746be-177a-4615-be53-4f09b8ffcaf3 2024-09-18T10:47:13.074 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 a21746be-177a-4615-be53-4f09b8ffcaf3 2024-09-18T10:47:13.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a21746be-177a-4615-be53-4f09b8ffcaf3' 2024-09-18T10:47:13.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:47:13.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxr+pmlT42BRAAJaluhDCBLIjB2IQv8XT6Wg== 2024-09-18T10:47:13.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxr+pmlT42BRAAJaluhDCBLIjB2IQv8XT6Wg=="}' 2024-09-18T10:47:13.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a21746be-177a-4615-be53-4f09b8ffcaf3 -i td/test-erasure-eio/1/new.json 2024-09-18T10:47:13.622 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:47:13.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T10:47:13.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxr+pmlT42BRAAJaluhDCBLIjB2IQv8XT6Wg== --osd-uuid a21746be-177a-4615-be53-4f09b8ffcaf3 2024-09-18T10:47:13.670 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.662+0000 7f4408b66540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:13.674 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.666+0000 7f4408b66540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:13.675 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.666+0000 7f4408b66540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:13.676 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.666+0000 7f4408b66540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:47:13.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.666+0000 7f4408b66540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:47:13.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.666+0000 7f4408b66540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:47:13.775 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:13.766+0000 7f4408b66540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T10:47:19.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T10:47:19.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:47:19.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T10:47:19.812 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T10:47:19.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:47:20.360 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T10:47:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T10:47:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:47:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:47:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:47:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:47:20.396 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:20.386+0000 7f0f2f157540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:20.410+0000 7f0f2f157540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:20.420 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:20.410+0000 7f0f2f157540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T10:47:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:47:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T10:47:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:47:20.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:47:20.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:20.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:47:20.778 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:47:20.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:20.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:20.988 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:20.978+0000 7f0f2f157540 -1 Falling back to public interface 2024-09-18T10:47:21.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:22.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:22.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:22.207 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:47:22.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:47:22.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:22.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:22.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:23.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:23.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:23.638 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:47:23.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:47:23.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:23.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:23.687 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:23.678+0000 7f0f2f157540 -1 osd.1 0 log_to_monitors true 2024-09-18T10:47:24.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:25.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:25.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:25.172 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:47:25.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:47:25.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:25.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:25.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:26.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:26.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:26.711 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:47:26.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:47:26.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:26.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:27.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:28.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:28.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:28.111 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:47:28.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:47:28.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:28.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:28.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:29.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:29.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:29.566 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:47:29.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:47:29.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:29.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:30.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:31.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:31.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:31.003 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:47:31.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:47:31.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:31.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:31.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:32.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:32.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:32.590 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:47:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:47:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:33.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:34.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:34.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:34.018 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:47:34.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:47:34.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:34.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:34.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:35.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:35.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:35.494 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:47:35.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:47:35.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:35.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:36.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:37.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:37.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:37.092 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:47:37.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:47:37.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:37.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:37.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:38.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:38.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:38.550 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:47:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:47:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:39.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:40.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:40.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:40.002 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:47:40.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:47:40.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:40.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:40.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:41.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:41.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:41.447 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:47:41.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:47:41.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:41.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:42.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:43.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:43.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:43.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:47:43.008 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:47:43.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:43.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:43.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:44.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:44.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:44.811 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:47:44.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:47:44.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:44.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:45.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:46.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:46.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:46.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:47:46.578 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:47:46.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:46.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:47.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:48.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:48.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:48.040 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:47:48.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:47:48.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:48.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:47:48.530 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/82484,v1:127.0.0.1:6811/82484] [v2:127.0.0.1:6812/82484,v1:127.0.0.1:6813/82484] exists,up a21746be-177a-4615-be53-4f09b8ffcaf3 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T10:47:48.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:47:48.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:47:48.536 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:47:48.536 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:47:48.536 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:47:48.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T10:47:48.538 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:47:48.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 2024-09-18T10:47:48.539 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 2024-09-18T10:47:48.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12' 2024-09-18T10:47:48.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:47:48.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUr+pmLAMaIRAAVoCo2nGpCrDfSiD0aRjzSg== 2024-09-18T10:47:48.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUr+pmLAMaIRAAVoCo2nGpCrDfSiD0aRjzSg=="}' 2024-09-18T10:47:48.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 -i td/test-erasure-eio/2/new.json 2024-09-18T10:47:49.110 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:47:49.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T10:47:49.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUr+pmLAMaIRAAVoCo2nGpCrDfSiD0aRjzSg== --osd-uuid e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 2024-09-18T10:47:49.156 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.146+0000 7fd85e52e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.166+0000 7fd85e52e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.166+0000 7fd85e52e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.166+0000 7fd85e52e540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:47:49.177 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.166+0000 7fd85e52e540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:47:49.177 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.166+0000 7fd85e52e540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:47:49.265 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:49.254+0000 7fd85e52e540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T10:47:55.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T10:47:55.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:47:55.636 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T10:47:55.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T10:47:55.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:47:56.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T10:47:56.195 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:47:56.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:47:56.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:47:56.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:47:56.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:47:56.228 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:56.218+0000 7f36d089d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:56.230 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:56.222+0000 7f36d089d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:56.233 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:56.222+0000 7f36d089d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:47:56.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T10:47:56.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:47:56.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:47:56.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:47:56.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:47:56.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:56.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:47:56.644 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:47:56.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:56.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:47:56.788 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:56.778+0000 7f36d089d540 -1 Falling back to public interface 2024-09-18T10:47:57.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:58.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:58.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:58.092 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:47:58.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:47:58.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:58.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:47:58.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:47:59.415 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:47:59.405+0000 7f36d089d540 -1 osd.2 0 log_to_monitors true 2024-09-18T10:47:59.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:47:59.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:47:59.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:47:59.566 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:47:59.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:47:59.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:00.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:01.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:01.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:01.125 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:48:01.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:48:01.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:01.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:01.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:02.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:02.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:02.595 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:48:02.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:48:02.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:02.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:03.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:04.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:04.184 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:48:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:48:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:04.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:05.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:05.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:05.679 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:48:05.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:48:05.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:05.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:06.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:07.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:07.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:07.125 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:48:07.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:48:07.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:07.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:07.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:08.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:08.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:08.548 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:48:08.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:48:08.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:08.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:09.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:10.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:10.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:10.004 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:48:10.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:48:10.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:10.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:10.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:11.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:11.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:11.527 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:48:11.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:48:11.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:11.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:12.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:13.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:13.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:13.138 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:48:13.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:48:13.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:13.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:14.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:15.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:15.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:15.132 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:48:15.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:48:15.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:15.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:15.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:16.805 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:48:16.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:48:16.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:16.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:17.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:18.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:18.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:18.723 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:48:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:48:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:19.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:20.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:20.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:20.945 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:48:20.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:48:20.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:20.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:21.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:22.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:22.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:22.953 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:48:22.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:48:22.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:22.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:48:23.760 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/84288,v1:127.0.0.1:6819/84288] [v2:127.0.0.1:6820/84288,v1:127.0.0.1:6821/84288] exists,up e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 2024-09-18T10:48:23.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:48:23.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:48:23.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:48:23.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:48:23.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:48:23.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:48:23.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:48:23.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:48:23.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:48:23.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:48:23.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:48:23.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T10:48:23.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:48:23.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5cc5df32-603d-42e9-8739-661f66c93c1b 2024-09-18T10:48:23.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5cc5df32-603d-42e9-8739-661f66c93c1b' 2024-09-18T10:48:23.768 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 5cc5df32-603d-42e9-8739-661f66c93c1b 2024-09-18T10:48:23.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:48:23.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3r+pmkiDVLhAAH55NroqyBBPqKhYc3yfcxw== 2024-09-18T10:48:23.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3r+pmkiDVLhAAH55NroqyBBPqKhYc3yfcxw=="}' 2024-09-18T10:48:23.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5cc5df32-603d-42e9-8739-661f66c93c1b -i td/test-erasure-eio/3/new.json 2024-09-18T10:48:24.610 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:48:24.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T10:48:24.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3r+pmkiDVLhAAH55NroqyBBPqKhYc3yfcxw== --osd-uuid 5cc5df32-603d-42e9-8739-661f66c93c1b 2024-09-18T10:48:24.667 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.657+0000 7f2220dd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:48:24.670 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.661+0000 7f2220dd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:48:24.672 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.661+0000 7f2220dd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:48:24.673 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.661+0000 7f2220dd1540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:48:24.673 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.661+0000 7f2220dd1540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:48:24.673 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.661+0000 7f2220dd1540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:48:24.774 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:24.765+0000 7f2220dd1540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T10:48:30.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T10:48:30.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:48:30.752 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T10:48:30.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T10:48:30.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:48:31.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T10:48:31.318 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T10:48:31.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:48:31.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:48:31.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:48:31.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:48:31.344 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:31.333+0000 7f626930c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:48:31.363 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:31.353+0000 7f626930c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:48:31.364 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:31.353+0000 7f626930c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:48:31.664 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:31.653+0000 7f626930c540 -1 Falling back to public interface 2024-09-18T10:48:31.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T10:48:31.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:48:31.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T10:48:31.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:48:31.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:48:31.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:31.742 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:48:31.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:48:31.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:31.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:32.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:33.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:33.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:33.159 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:48:33.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:48:33.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:33.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:33.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:34.370 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:48:34.361+0000 7f626930c540 -1 osd.3 0 log_to_monitors true 2024-09-18T10:48:34.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:34.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:34.597 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:48:34.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:48:34.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:34.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:35.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:36.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:36.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:36.271 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:48:36.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:48:36.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:36.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:37.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:38.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:38.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:38.003 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:48:38.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:48:38.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:38.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:38.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:39.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:39.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:39.423 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:48:39.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:48:39.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:39.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:39.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:40.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:40.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:40.848 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:48:40.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:48:40.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:40.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:41.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:42.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:42.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:42.333 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:48:42.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:48:42.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:42.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:42.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:43.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:43.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:43.751 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:48:43.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:48:43.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:43.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:44.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:45.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:45.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:45.202 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:48:45.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:48:45.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:45.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:45.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:46.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:46.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:46.691 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:48:46.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:48:46.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:46.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:47.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:48.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:48.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:48.204 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:48:48.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:48:48.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:48.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:48.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:49.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:49.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:49.989 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:48:49.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:48:49.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:49.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:50.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:51.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:51.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:51.443 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:48:51.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:48:51.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:51.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:52.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:53.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:53.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:53.280 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:48:53.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:48:53.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:53.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:54.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:55.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:55.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:55.085 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:48:55.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:48:55.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:55.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:55.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:56.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:56.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:56.806 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:48:56.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:48:56.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:56.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:57.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:48:58.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:48:58.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:48:58.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:48:58.745 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:48:58.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:48:58.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:48:59.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:00.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:00.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:00.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:49:00.166 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:49:00.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:00.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:49:00.877 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/86037,v1:127.0.0.1:6827/86037] [v2:127.0.0.1:6828/86037,v1:127.0.0.1:6829/86037] exists,up 5cc5df32-603d-42e9-8739-661f66c93c1b 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:49:00.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:49:00.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:49:00.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:49:00.882 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:49:00.882 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:49:00.882 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:49:00.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:49:00.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T10:49:00.884 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:49:00.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84d9b69b-444a-481c-b665-e0ab1a3f3976 2024-09-18T10:49:00.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 84d9b69b-444a-481c-b665-e0ab1a3f3976' 2024-09-18T10:49:00.885 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 84d9b69b-444a-481c-b665-e0ab1a3f3976 2024-09-18T10:49:00.886 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:49:00.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcsOpmRaHoNRAAeu72XqPfb6v/gSgBA9WE2A== 2024-09-18T10:49:00.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcsOpmRaHoNRAAeu72XqPfb6v/gSgBA9WE2A=="}' 2024-09-18T10:49:00.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84d9b69b-444a-481c-b665-e0ab1a3f3976 -i td/test-erasure-eio/4/new.json 2024-09-18T10:49:01.632 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:49:01.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T10:49:01.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcsOpmRaHoNRAAeu72XqPfb6v/gSgBA9WE2A== --osd-uuid 84d9b69b-444a-481c-b665-e0ab1a3f3976 2024-09-18T10:49:01.690 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.680+0000 7f5c61ae6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:49:01.716 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.708+0000 7f5c61ae6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:49:01.718 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.712+0000 7f5c61ae6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:49:01.719 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.712+0000 7f5c61ae6540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:49:01.719 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.712+0000 7f5c61ae6540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:49:01.719 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.712+0000 7f5c61ae6540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:49:01.816 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:01.808+0000 7f5c61ae6540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T10:49:08.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T10:49:08.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:49:08.037 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T10:49:08.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T10:49:08.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:49:08.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T10:49:08.582 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T10:49:08.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:49:08.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:49:08.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:49:08.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:49:08.614 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:08.608+0000 7f304b331540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:49:08.617 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:08.608+0000 7f304b331540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:49:08.618 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:08.612+0000 7f304b331540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:08.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:49:08.945 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:49:08.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:08.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:09.168 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:09.160+0000 7f304b331540 -1 Falling back to public interface 2024-09-18T10:49:09.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:10.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:10.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:10.462 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:49:10.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:49:10.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:10.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:10.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:11.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:11.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:11.882 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:49:11.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:49:11.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:11.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:12.013 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:49:12.004+0000 7f304b331540 -1 osd.4 0 log_to_monitors true 2024-09-18T10:49:12.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:13.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:13.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:13.341 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:49:13.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:49:13.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:13.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:13.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:14.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:14.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:14.855 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:49:14.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:49:14.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:14.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:15.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:16.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:16.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:16.431 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:49:16.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:49:16.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:16.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:16.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:17.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:17.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:17.897 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:49:17.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:49:17.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:17.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:18.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:19.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:19.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:19.331 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:49:19.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:49:19.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:19.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:19.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:20.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:20.803 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:49:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:49:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:20.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:21.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:22.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:22.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:22.236 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:49:22.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:49:22.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:22.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:22.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:23.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:23.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:23.932 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:49:23.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:49:23.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:23.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:24.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:25.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:25.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:25.348 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:49:25.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:49:25.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:25.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:25.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:26.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:26.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:26.909 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:49:26.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:49:26.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:26.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:27.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:28.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:28.896 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:49:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:49:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:29.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:30.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:30.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:30.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:49:30.920 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:49:30.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:30.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:31.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:32.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:32.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:32.946 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:49:32.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:49:32.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:32.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:34.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:35.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:35.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:35.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:49:35.003 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:49:35.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:35.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:35.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:36.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:36.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:36.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:49:36.665 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:49:36.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:36.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:37.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:49:38.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:49:38.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:49:38.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:49:38.160 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:49:38.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:49:38.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:49:38.627 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/87838,v1:127.0.0.1:6835/87838] [v2:127.0.0.1:6836/87838,v1:127.0.0.1:6837/87838] exists,up 84d9b69b-444a-481c-b665-e0ab1a3f3976 2024-09-18T10:49:38.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:49:38.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:49:38.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:49:38.628 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T10:49:38.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:49:38.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:49:38.629 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:49:38.629 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:49:38.629 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:49:38.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T10:49:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T10:49:38.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T10:49:38.710 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:49:38.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T10:49:38.725 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:46:46.715+0000 7f44b104a540 0 load: jerasure load: lrc 2024-09-18T10:49:38.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:49:38.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T10:49:38.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-18T10:49:39.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T10:49:39.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T10:49:41.269 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T10:49:41.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:49:42.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T10:49:42.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:49:42.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:49:42.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:49:42.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:49:42.285 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:49:42.286 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:49:42.286 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:49:42.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:49:42.287 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:49:42.287 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:49:42.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:49:42.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:49:42.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:49:42.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:49:42.494 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:49:42.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:49:42.899 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:49:42.899 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:49:42.899 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:49:42.900 INFO:tasks.workunit.client.0.smithi174.stderr:4' 2024-09-18T10:49:42.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:49:42.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:49:42.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:49:43.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803814 2024-09-18T10:49:43.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803814 2024-09-18T10:49:43.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814' 2024-09-18T10:49:43.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:49:43.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:49:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574877 2024-09-18T10:49:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574877 2024-09-18T10:49:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574877' 2024-09-18T10:49:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:49:43.318 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:49:43.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378652 2024-09-18T10:49:43.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378652 2024-09-18T10:49:43.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574877 2-81604378652' 2024-09-18T10:49:43.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:49:43.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:49:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051607 2024-09-18T10:49:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051607 2024-09-18T10:49:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574877 2-81604378652 3-124554051607' 2024-09-18T10:49:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:49:43.743 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:49:43.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691850 2024-09-18T10:49:43.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691850 2024-09-18T10:49:43.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574877 2-81604378652 3-124554051607 4-171798691850' 2024-09-18T10:49:43.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:49:43.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803814 2024-09-18T10:49:43.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:49:43.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:49:43.895 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803814 2024-09-18T10:49:43.895 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:49:43.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803814 2024-09-18T10:49:43.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-18T10:49:43.897 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803814 2024-09-18T10:49:43.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:49:44.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803814 -lt 25769803814 2024-09-18T10:49:44.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:49:44.314 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574877 2024-09-18T10:49:44.314 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:49:44.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:49:44.317 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:49:44.318 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574877 2024-09-18T10:49:44.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574877 2024-09-18T10:49:44.319 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574877 2024-09-18T10:49:44.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-09-18T10:49:44.319 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:49:44.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574875 -lt 55834574877 2024-09-18T10:49:44.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T10:49:45.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T10:49:45.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:49:46.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574877 -lt 55834574877 2024-09-18T10:49:46.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:49:46.200 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378652 2024-09-18T10:49:46.200 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:49:46.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:49:46.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378652 2024-09-18T10:49:46.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:49:46.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378652 2024-09-18T10:49:46.205 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378652 2024-09-18T10:49:46.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378652' 2024-09-18T10:49:46.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:49:46.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378653 -lt 81604378652 2024-09-18T10:49:46.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:49:46.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051607 2024-09-18T10:49:46.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:49:46.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:49:46.611 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051607 2024-09-18T10:49:46.611 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:49:46.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051607 2024-09-18T10:49:46.613 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051607 2024-09-18T10:49:46.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051607' 2024-09-18T10:49:46.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:49:47.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051608 -lt 124554051607 2024-09-18T10:49:47.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:49:47.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:49:47.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691850 2024-09-18T10:49:47.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:49:47.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691850 2024-09-18T10:49:47.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:49:47.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691850 2024-09-18T10:49:47.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691850' 2024-09-18T10:49:47.053 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 171798691850 2024-09-18T10:49:47.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:49:47.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691850 -lt 171798691850 2024-09-18T10:49:47.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:49:47.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:49:47.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:49:48.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T10:49:48.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:49:48.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:49:48.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:49:48.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:49:48.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:49:48.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:49:48.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:49:48.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T10:49:48.433 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:49:48.433 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:49:48.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:49:48.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T10:49:48.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:49:48.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:49:48.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:43.116789+0000 0'0 48:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-18T10:49:40.728480+0000 0'0 2024-09-18T10:49:40.728480+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:52:13.342034+0000 0 0 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.956915+0000 0'0 48:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:16:27.738440+0000 0 0 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.853604+0000 0'0 48:107 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:00:31.680808+0000 0 0 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.486099+0000 0'0 48:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T11:37:23.822730+0000 0 0 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:48:23.632326+0000 0'0 48:46 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:39:47.726841+0000 0 0 2024-09-18T10:49:49.349 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T10:49:49.350 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T10:49:49.352 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T10:49:49.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-18T10:49:49.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T10:49:49.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T10:49:49.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T10:49:49.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:49.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-09-18T10:49:49.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-09-18T10:49:49.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-18T10:49:49.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-18T10:49:49.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T10:49:49.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-09-18T10:49:49.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:49:49.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-09-18T10:49:49.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:49:49.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T10:49:49.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T10:49:49.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:49:50.414 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T10:49:50.414 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:49:50.414 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:49:50.415 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:49:50.415 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T10:49:50.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-09-18T10:49:50.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-09-18T10:49:50.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-09-18T10:49:50.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T10:49:50.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:49:50.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:49:50.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:49:50.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:49:50.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:49:50.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:49:50.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-09-18T10:49:51.309 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-18T10:49:51.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-09-18T10:49:52.122 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-18T10:49:52.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-18T10:49:52.517 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T10:49:52.517 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-09-18T10:49:43.116789+0000 49'1 49:13 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-18T10:49:40.728480+0000 0'0 2024-09-18T10:49:40.728480+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:52:13.342034+0000 0 0 2024-09-18T10:49:52.517 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.956915+0000 0'0 48:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:16:27.738440+0000 0 0 2024-09-18T10:49:52.517 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.853604+0000 0'0 50:109 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:00:31.680808+0000 0 0 2024-09-18T10:49:52.517 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.486099+0000 0'0 48:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T11:37:23.822730+0000 0 0 2024-09-18T10:49:52.517 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:48:23.632326+0000 0'0 48:46 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T21:39:47.726841+0000 0 0 2024-09-18T10:49:52.518 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T10:49:52.518 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T10:49:52.520 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T10:49:52.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-18T10:49:52.531 INFO:tasks.workunit.client.0.smithi174.stderr:4+0 records in 2024-09-18T10:49:52.531 INFO:tasks.workunit.client.0.smithi174.stderr:4+0 records out 2024-09-18T10:49:52.531 INFO:tasks.workunit.client.0.smithi174.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000287826 s, 14.2 MB/s 2024-09-18T10:49:52.531 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-09-18T10:49:52.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:52.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:52.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:52.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:52.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:52.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:53.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:53.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:53.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:53.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:54.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:54.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:54.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:54.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:54.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:54.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:55.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:55.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:55.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:55.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:55.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:55.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:55.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:55.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:56.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:56.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:56.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:56.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:56.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:56.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:56.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:56.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:57.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:57.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:57.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:57.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:57.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:57.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:57.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:57.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:58.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:58.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:58.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:58.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:58.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:58.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:58.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:58.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:59.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:59.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:59.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:59.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-18T10:49:59.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:49:59.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:00.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:00.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:00.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:00.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:00.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:00.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:00.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:00.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:01.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:01.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:01.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:01.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:01.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:01.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:01.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:01.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:02.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:02.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:02.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:02.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:02.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:02.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:03.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:03.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:03.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:03.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:03.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:03.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:03.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:03.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:04.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:04.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:04.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:04.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:04.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:04.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:04.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:04.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:05.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:05.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:05.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:05.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:05.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:05.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:05.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:06.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:06.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:06.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:06.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:06.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:06.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:06.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:06.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:07.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:07.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:07.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:07.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:07.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:07.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:08.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:08.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:08.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:08.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:08.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:08.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:08.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:08.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:09.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:09.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:09.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:09.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:09.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:09.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:09.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:09.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:10.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:10.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:10.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:10.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:10.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:10.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:10.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:10.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:11.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:11.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:11.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:11.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:11.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:11.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:12.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:12.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:12.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:12.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:12.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:12.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:12.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:12.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:13.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:13.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:13.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:13.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:13.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:13.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:13.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:13.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:14.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:14.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:14.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:14.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:14.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:14.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:15.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:15.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:15.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:15.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:15.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:15.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:15.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:15.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:16.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:16.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:16.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:16.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:16.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:16.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:16.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:16.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:17.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:17.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:17.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:17.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:17.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:17.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:17.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:17.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:18.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:18.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:18.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:18.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:18.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:18.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:18.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:18.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:19.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:19.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:19.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:19.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:19.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:19.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:20.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:20.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:20.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:20.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:20.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:20.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:20.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:20.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:21.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:21.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:21.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:21.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:21.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:21.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:21.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:21.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:22.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:22.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:22.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:22.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:22.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:22.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:22.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:22.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:23.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:23.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:23.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:23.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:23.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:23.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:24.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:24.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:24.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:24.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:24.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:24.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:24.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:24.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:25.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:25.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:25.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:25.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:25.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:25.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:25.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:25.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:26.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:26.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:26.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:26.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:26.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:26.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:26.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:26.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:27.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:27.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:27.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:27.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:27.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:27.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:27.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:27.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:28.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:28.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:28.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:28.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:28.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:28.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:29.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:29.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:29.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:29.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:29.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:29.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:29.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:29.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:30.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:30.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:30.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:30.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:30.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:30.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:31.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:31.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:31.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:31.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:31.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:31.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:31.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:31.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:32.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:32.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:32.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:32.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:32.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:32.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:32.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:32.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:33.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:33.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:33.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:33.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:33.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:33.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:33.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:33.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:34.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:34.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:34.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:34.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:34.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:34.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:35.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:35.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:35.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:35.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:35.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:35.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:35.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:35.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:36.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:36.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:36.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:36.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:36.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:36.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:36.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:36.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:37.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:37.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:37.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:37.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:37.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:37.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:37.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:37.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:38.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:38.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:38.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:38.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:38.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:38.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:39.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:39.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:39.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:39.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:39.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:39.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:39.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:39.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:40.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:40.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:40.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:40.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:40.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:40.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:41.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:41.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:41.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:41.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:41.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:41.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:41.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:41.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:42.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:42.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:42.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:42.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:42.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:42.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:42.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:43.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:43.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:43.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:43.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:43.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:43.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:43.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:43.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:44.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:44.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:44.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:44.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:44.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:44.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:44.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:44.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:45.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:45.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:45.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:45.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:46.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:46.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:46.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:46.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:46.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:46.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:46.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:46.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:47.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:47.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:47.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:47.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:47.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:47.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:47.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:47.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:48.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:48.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:48.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:48.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:48.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:48.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:49.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:49.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:49.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:49.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:49.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:49.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:50.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:50.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:50.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:50.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:50.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:50.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:50.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:50.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:51.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:51.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:51.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:51.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:51.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:51.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:51.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:51.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:52.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:52.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:52.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:52.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:52.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:52.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:52.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:52.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:53.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:53.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:53.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:53.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:53.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:53.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:53.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:53.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:54.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:54.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:54.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:54.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:54.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:54.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:54.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:54.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:55.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:55.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:55.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:55.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:55.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:55.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:55.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:55.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:56.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:56.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:56.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:56.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:56.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:56.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:57.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:57.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:57.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:57.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:57.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:57.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:57.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:57.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:58.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:58.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:58.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:58.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:58.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:58.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:58.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:58.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:59.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:59.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:59.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:59.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:59.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:59.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-09-18T10:50:59.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:50:59.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:00.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:00.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:00.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:00.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:00.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:00.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:00.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:00.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:01.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:01.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:01.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:01.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:01.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:01.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:02.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:02.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:02.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:02.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:02.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:02.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:02.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:03.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:03.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:03.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:03.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:03.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:03.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:03.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:03.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:04.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:04.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:04.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:04.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:04.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:04.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:04.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:04.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:05.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:05.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:05.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:05.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:05.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:05.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:05.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:05.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:06.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:06.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:06.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:06.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:06.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:06.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:07.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:07.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:07.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:07.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:07.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:07.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:07.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:07.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:08.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:08.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:08.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:08.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:08.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:08.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:08.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:08.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:09.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:09.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:09.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:09.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:09.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:09.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:10.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:10.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:10.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:10.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:10.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:10.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:10.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:10.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:11.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:11.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:11.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:11.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:11.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:11.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:11.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:11.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:12.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:12.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:12.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:51:12.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-09-18T10:51:12.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-09-18T10:51:12.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T10:51:12.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T10:51:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:51:12.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj250 2024-09-18T10:51:12.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:51:12.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-18T10:51:12.721 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-18T10:51:12.721 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:51:13.166 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:2147483647' 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T10:51:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T10:51:13.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T10:51:13.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T10:51:13.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T10:51:13.170 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T10:51:13.170 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T10:51:13.170 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T10:51:13.171 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:51:13.171 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:51:13.171 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:51:13.171 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T10:51:13.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T10:51:13.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T10:51:13.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T10:51:13.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T10:51:13.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T10:51:13.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T10:51:13.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T10:51:13.262 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:51:13.262 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:51:13.262 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:51:13.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T10:51:13.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T10:51:13.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-18T10:51:13.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj250 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-18T10:51:13.357 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-09-18T10:51:13.358 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:51:13.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T10:51:13.767 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:51:13.767 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:51:13.767 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:51:13.767 INFO:tasks.workunit.client.0.smithi174.stderr:2147483647' 2024-09-18T10:51:13.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-18T10:51:13.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T10:51:13.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T10:51:13.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T10:51:13.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T10:51:13.770 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T10:51:13.772 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T10:51:13.772 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T10:51:13.772 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T10:51:13.773 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:51:13.773 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:51:13.773 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:51:13.774 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T10:51:13.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T10:51:13.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T10:51:13.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T10:51:13.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T10:51:13.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T10:51:13.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T10:51:13.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T10:51:13.865 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:51:13.865 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:51:13.865 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:51:13.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T10:51:13.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T10:51:13.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-09-18T10:51:13.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T10:51:13.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T10:51:13.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:51:13.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:51:13.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:51:13.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:51:13.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:51:13.961 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:51:13.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:51:13.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:51:13.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:51:13.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:51:13.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:51:13.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T10:51:13.964 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:51:13.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T10:51:13.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:51:13.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T10:51:13.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T10:51:13.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:51:13.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:51:13.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:51:13.988 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:13.979+0000 7fbf00f52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:51:13.990 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:13.983+0000 7fbf00f52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:51:13.992 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:13.983+0000 7fbf00f52540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:51:14.291 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:14.283+0000 7fbf00f52540 -1 Falling back to public interface 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:51:14.390 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:51:14.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:51:14.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:51:14.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:51:14.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:51:15.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:51:15.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:51:15.801 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:51:15.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:51:15.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:51:15.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:51:16.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:51:17.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:51:17.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:51:17.223 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:51:17.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:51:17.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:51:17.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:51:17.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:51:17.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:17.671+0000 7fbf00f52540 -1 osd.2 49 log_to_monitors true 2024-09-18T10:51:18.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:51:18.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:51:18.676 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:51:18.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:51:18.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:51:18.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:51:19.178 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up out weight 0 up_from 56 up_thru 19 down_at 50 last_clean_interval [19,49) [v2:127.0.0.1:6818/95630,v1:127.0.0.1:6819/95630] [v2:127.0.0.1:6820/95630,v1:127.0.0.1:6821/95630] exists,up e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 2024-09-18T10:51:19.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:51:19.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:51:19.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:51:19.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-09-18T10:51:20.091 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-18T10:51:20.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-09-18T10:51:35.101 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-09-18T10:51:35.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:35.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:35.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:35.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:35.105 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:35.105 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:35.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:35.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:35.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:35.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:35.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:35.526 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:35.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:36.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:36.528 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:36.528 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:36.528 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:36.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:36.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:36.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:36.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:36.942 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:36.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:37.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:37.944 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:37.944 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:37.944 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:37.945 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:37.945 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:38.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:38.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:38.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:38.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:38.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:38.353 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:38.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:39.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:39.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:39.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:39.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:39.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:39.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:39.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:39.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:39.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:39.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:39.768 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:39.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:39.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:40.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:40.770 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:40.770 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:40.770 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:40.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:40.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:41.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:41.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:41.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:41.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:41.163 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:41.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:41.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:42.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:42.165 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:42.165 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:42.165 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:42.165 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:42.166 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:42.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:42.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:42.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:42.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:42.567 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:42.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:42.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:43.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:43.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:43.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:43.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:43.570 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:43.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:43.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:43.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:43.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:43.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:43.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:43.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:43.984 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:44.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:44.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:44.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:44.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:44.988 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:44.988 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:45.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:45.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:45.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:45.374 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:45.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:45.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:45.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:46.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:46.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:46.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:46.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:46.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:46.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:46.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:46.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:46.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:46.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:46.823 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:46.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:46.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:47.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:47.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:47.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:47.826 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:47.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:47.827 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:48.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:48.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:48.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:48.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:48.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:48.234 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:48.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:49.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:49.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:49.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:49.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:49.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:49.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:49.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:49.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:49.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:49.666 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:49.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:49.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:49.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:50.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:50.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:50.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:50.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:50.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:50.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:51.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:51.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:51.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:51.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-09-18T10:51:51.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:51.058 INFO:tasks.workunit.client.0.smithi174.stdout:active+undersized+degraded+remapped+backfilling 2024-09-18T10:51:51.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-09-18T10:51:52.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-09-18T10:51:52.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-09-18T10:51:52.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T10:51:52.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T10:51:52.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T10:51:52.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T10:51:52.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-18T10:51:52.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-09-18T10:51:52.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-18T10:51:52.473 INFO:tasks.workunit.client.0.smithi174.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-18T10:51:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-09-18T10:51:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-09-18T10:51:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-18T10:51:52.865 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T10:51:52.865 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-18T10:51:45.615669+0000 54'301 62:902 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-09-18T10:49:40.728480+0000 0'0 2024-09-18T10:49:40.728480+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:27:49.738686+0000 0 0 2024-09-18T10:51:52.865 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:51:21.350594+0000 0'0 61:53 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:51:15.329128+0000 0 0 2024-09-18T10:51:52.866 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.853604+0000 0'0 61:118 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:00:31.680808+0000 0 0 2024-09-18T10:51:52.866 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.486099+0000 0'0 61:26 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T11:37:23.822730+0000 0 0 2024-09-18T10:51:52.866 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:51:21.351072+0000 0'0 61:84 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:03:40.303621+0000 0 0 2024-09-18T10:51:52.866 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T10:51:52.866 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T10:51:52.868 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T10:51:52.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T10:51:52.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:51:52.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:51:52.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:51:52.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:51:52.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:51:53.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:51:53.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-09-18T10:51:58.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-09-18T10:51:54.189759+0000 54'301 64:917 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-18T10:49:40.728480+0000 0'0 2024-09-18T10:49:40.728480+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:13:37.378373+0000 0 0 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:51:21.350594+0000 0'0 61:53 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:51:15.329128+0000 0 0 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.853604+0000 0'0 64:121 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:00:31.680808+0000 0 0 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T10:49:40.486099+0000 0'0 61:26 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T11:37:23.822730+0000 0 0 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-18T10:51:54.033246+0000 0'0 64:94 [1,0] 1 [1,0] 1 0'0 2024-09-18T10:46:38.367551+0000 0'0 2024-09-18T10:46:38.367551+0000 0 0 periodic scrub scheduled @ 2024-09-19T18:56:28.991894+0000 0 0 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T10:51:58.603 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T10:51:58.606 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T10:51:58.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-18T10:51:58.758 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "num_missing": 1, 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "num_unfound": 1, 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": [ 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "oid": { 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "oid": "obj250", 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "key": "", 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "snapid": -2, 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": 2249616407, 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "max": 0, 2024-09-18T10:51:58.759 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 2, 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "namespace": "" 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "need": "54'251", 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "have": "0'0", 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "none", 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "locations": [ 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "NotRecovering", 2024-09-18T10:51:58.760 INFO:tasks.workunit.client.0.smithi174.stdout: "available_might_have_unfound": true, 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: "might_have_unfound": [ 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "2(4)", 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "osd is down" 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout: "more": false 2024-09-18T10:51:58.761 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-18T10:51:58.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-09-18T10:51:58.904 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-18T10:51:58.904 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trimq": "[]", 2024-09-18T10:51:58.904 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trimq_len": 0, 2024-09-18T10:51:58.904 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-18T10:51:58.904 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 65, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.905 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_recovery_backfill": [ 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "info": { 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s0", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "54'301", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "54'250", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "54'200", 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 301, 2024-09-18T10:51:58.906 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 47, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 47, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 63, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 47, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 63, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 63, 2024-09-18T10:51:58.907 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 47, 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "54'301", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 918, 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 65, 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T10:51:54.661521+0000", 2024-09-18T10:51:58.908 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T10:51:54.189759+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T10:51:54.661521+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T10:51:54.661521+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T10:49:49.530767+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T10:51:54.160864+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T10:51:54.160864+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T10:51:54.661521+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T10:51:53.809771+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T10:51:53.809262+0000", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 63, 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "54'200", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "54'200", 2024-09-18T10:51:58.909 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 47, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 200, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T10:51:58.910 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T17:13:37.378373+0000", 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 1232896, 2024-09-18T10:51:58.911 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 301, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 1505, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 1, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 1, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 303, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 1, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 301, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 301, 2024-09-18T10:51:58.912 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 1204, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T10:51:58.913 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.914 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T10:51:58.915 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 300 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),4(2)", 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 1 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T10:51:58.916 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "peer_info": [ 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "0(3)", 2024-09-18T10:51:58.917 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s3", 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "54'301", 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "54'301", 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "54'200", 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 301, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 47, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 47, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 63, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 47, 2024-09-18T10:51:58.918 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 63, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 63, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 47, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "54'301", 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 907, 2024-09-18T10:51:58.919 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 63, 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T10:49:49.530767+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T10:49:51.279364+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T10:49:51.279052+0000", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 63, 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "54'200", 2024-09-18T10:51:58.920 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "54'200", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 47, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 200, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T10:51:58.921 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T20:25:18.526094+0000", 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 1232896, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 301, 2024-09-18T10:51:58.922 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 1505, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 301, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 301, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 301, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 1204, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T10:51:58.923 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T10:51:58.924 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.925 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T10:51:58.926 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 301 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T10:51:58.927 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "1(1)", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s1", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "54'301", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "54'301", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "54'200", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 301, 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T10:51:58.928 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 47, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 47, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 63, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 47, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 63, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 63, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 47, 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.929 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "54'301", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 328, 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 54, 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.930 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T10:49:49.530767+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T10:49:51.279364+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T10:49:51.279052+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 63, 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "54'200", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "54'200", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 47, 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.931 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 200, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T10:51:58.932 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T20:25:18.526094+0000", 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 1232896, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 301, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 1505, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 1, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 301, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T10:51:58.933 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 301, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 301, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 1204, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T10:51:58.934 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T10:51:58.935 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.936 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T10:51:58.937 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 301 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T10:51:58.938 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "4(2)", 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s2", 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "54'301", 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "54'301", 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "54'200", 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 301, 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 47, 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 47, 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.939 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 63, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 47, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 63, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 63, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 47, 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T10:51:58.940 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "54'301", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 328, 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 54, 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T10:49:49.530767+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T10:49:51.599003+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T10:51:12.708547+0000", 2024-09-18T10:51:58.941 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T10:49:51.279364+0000", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T10:49:51.279052+0000", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 63, 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "54'200", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "54'200", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 47, 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 48, 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T10:49:40.728480+0000", 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T10:51:58.942 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 200, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T20:25:18.526094+0000", 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T10:51:58.943 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 1232896, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 301, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 1505, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 301, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 301, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T10:51:58.944 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 301, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 1204, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T10:51:58.945 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.946 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: 2147483647 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T10:51:58.947 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 301 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T10:51:58.948 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 64, 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_state": [ 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "Started/Primary/Active", 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "enter_time": "2024-09-18T10:51:53.809343+0000", 2024-09-18T10:51:58.949 INFO:tasks.workunit.client.0.smithi174.stdout: "might_have_unfound": [ 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "0(3)", 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "1(1)", 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "2(4)", 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "osd is down" 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.950 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "4(2)", 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_progress": { 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "backfill_targets": [], 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "waiting_on_backfill": [], 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill_started": "MIN", 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "backfill_info": { 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "begin": "MIN", 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "end": "MIN", 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": [] 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "peer_backfill_info": [], 2024-09-18T10:51:58.951 INFO:tasks.workunit.client.0.smithi174.stdout: "backfills_in_flight": [], 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering": [], 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_backend": { 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_ops": [], 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "read_ops": [] 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "Started", 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "enter_time": "2024-09-18T10:51:53.271997+0000" 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "scrubber": { 2024-09-18T10:51:58.952 INFO:tasks.workunit.client.0.smithi174.stdout: "active": false, 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "must_scrub": false, 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "must_deep_scrub": false, 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "must_repair": false, 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "need_auto": false, 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_reg_stamp": "2024-09-19T17:13:37.378373+0000", 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "schedule": "scrub scheduled @ 2024-09-19T17:13:37.378373+0000" 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout: "agent_state": {} 2024-09-18T10:51:58.953 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-18T10:51:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-18T10:51:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-09-18T10:51:59.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-18T10:51:59.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-09-18T10:51:59.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-09-18T10:51:59.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-09-18T10:51:59.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-18T10:51:59.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-09-18T10:51:59.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-09-18T10:51:59.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-09-18T10:51:59.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-09-18T10:51:59.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-18T10:51:59.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-09-18T10:51:59.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-09-18T10:51:59.510 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T10:51:59.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:51:59.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:51:59.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T10:51:59.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T10:51:59.515 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:51:59.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:51:59.515 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T10:51:59.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T10:51:59.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:51:59.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:51:59.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:51:59.537 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:59.531+0000 7f1855709540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:51:59.539 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:59.531+0000 7f1855709540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:51:59.541 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:51:59.535+0000 7f1855709540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:51:59.916 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:51:59.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:51:59.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:52:00.091 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:00.083+0000 7f1855709540 -1 Falling back to public interface 2024-09-18T10:52:00.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:01.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:01.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:01.333 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:52:01.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:52:01.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:01.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:52:01.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:02.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:02.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:02.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:52:02.730 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:52:02.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:02.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:52:03.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:03.997 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:03.991+0000 7f1855709540 -1 osd.2 62 log_to_monitors true 2024-09-18T10:52:04.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:04.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:04.129 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:52:04.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:52:04.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:04.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:52:04.768 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 67 up_thru 19 down_at 63 last_clean_interval [56,62) [v2:127.0.0.1:6818/96825,v1:127.0.0.1:6819/96825] [v2:127.0.0.1:6820/96825,v1:127.0.0.1:6821/96825] exists,up e1f6b1b1-aeea-4bcf-ad7c-ce32fda68a12 2024-09-18T10:52:04.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:52:04.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:52:04.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:52:04.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-18T10:52:09.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-09-18T10:52:09.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-18T10:52:10.043 INFO:tasks.workunit.client.0.smithi174.stderr:pg has no unfound objects 2024-09-18T10:52:10.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-09-18T10:52:10.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:52:10.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:52:10.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:52:10.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:52:10.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:52:10.055 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:52:10.055 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:52:10.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:52:10.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:52:10.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:52:10.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:52:10.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:52:10.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:52:10.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:52:10.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:52:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:52:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:52:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:52:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:52:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:4' 2024-09-18T10:52:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:52:10.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:52:10.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:52:10.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803845 2024-09-18T10:52:10.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803845 2024-09-18T10:52:10.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845' 2024-09-18T10:52:10.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:52:10.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:52:11.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574908 2024-09-18T10:52:11.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574908 2024-09-18T10:52:11.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-55834574908' 2024-09-18T10:52:11.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:52:11.012 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:52:11.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=287762808836 2024-09-18T10:52:11.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 287762808836 2024-09-18T10:52:11.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-287762808836' 2024-09-18T10:52:11.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:52:11.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:52:11.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051639 2024-09-18T10:52:11.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051639 2024-09-18T10:52:11.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-287762808836 3-124554051639' 2024-09-18T10:52:11.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:52:11.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:52:11.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691881 2024-09-18T10:52:11.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691881 2024-09-18T10:52:11.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-55834574908 2-287762808836 3-124554051639 4-171798691881' 2024-09-18T10:52:11.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:52:11.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803845 2024-09-18T10:52:11.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:52:11.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:52:11.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803845 2024-09-18T10:52:11.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:52:11.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803845 2024-09-18T10:52:11.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803845' 2024-09-18T10:52:11.542 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803845 2024-09-18T10:52:11.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:52:11.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803845 -lt 25769803845 2024-09-18T10:52:11.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:52:11.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574908 2024-09-18T10:52:11.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:52:11.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:52:11.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574908 2024-09-18T10:52:11.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:52:11.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574908 2024-09-18T10:52:11.983 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574908 2024-09-18T10:52:11.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574908' 2024-09-18T10:52:11.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:52:12.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574908 -lt 55834574908 2024-09-18T10:52:12.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:52:12.503 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-287762808836 2024-09-18T10:52:12.503 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:52:12.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:52:12.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-287762808836 2024-09-18T10:52:12.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:52:12.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=287762808836 2024-09-18T10:52:12.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 287762808836' 2024-09-18T10:52:12.508 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 287762808836 2024-09-18T10:52:12.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:52:12.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 287762808836 -lt 287762808836 2024-09-18T10:52:12.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:52:12.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051639 2024-09-18T10:52:12.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:52:12.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:52:12.971 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051639 2024-09-18T10:52:12.971 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:52:12.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051639 2024-09-18T10:52:12.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051639' 2024-09-18T10:52:12.973 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051639 2024-09-18T10:52:12.973 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:52:13.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051637 -lt 124554051639 2024-09-18T10:52:13.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T10:52:14.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T10:52:14.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:52:14.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051639 -lt 124554051639 2024-09-18T10:52:14.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:52:14.877 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:52:14.877 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-171798691881 2024-09-18T10:52:14.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:52:14.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-171798691881 2024-09-18T10:52:14.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:52:14.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691881 2024-09-18T10:52:14.882 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 171798691881 2024-09-18T10:52:14.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 171798691881' 2024-09-18T10:52:14.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:52:15.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691882 -lt 171798691881 2024-09-18T10:52:15.318 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:52:15.319 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:52:15.319 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:52:15.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:52:16.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T10:52:16.277 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:52:16.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:52:16.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:52:16.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T10:52:16.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:52:16.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:52:16.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-09-18T10:52:16.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:16.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-18T10:52:16.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-18T10:52:16.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:16.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:16.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-18T10:52:16.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-18T10:52:16.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:16.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:16.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-18T10:52:16.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-18T10:52:16.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:16.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:16.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-18T10:52:16.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-18T10:52:17.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-18T10:52:17.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-18T10:52:17.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-18T10:52:17.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-18T10:52:17.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-18T10:52:17.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-18T10:52:17.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-18T10:52:17.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-18T10:52:17.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-18T10:52:17.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-18T10:52:17.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-18T10:52:17.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-18T10:52:17.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-18T10:52:17.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-18T10:52:17.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-18T10:52:17.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-18T10:52:17.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-18T10:52:17.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-18T10:52:17.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-18T10:52:17.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:17.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:17.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-18T10:52:17.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-18T10:52:18.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-18T10:52:18.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-18T10:52:18.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-18T10:52:18.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-18T10:52:18.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-18T10:52:18.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-18T10:52:18.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-18T10:52:18.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-18T10:52:18.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-18T10:52:18.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-18T10:52:18.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-18T10:52:18.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-18T10:52:18.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-18T10:52:18.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-18T10:52:18.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-18T10:52:18.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-18T10:52:18.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-18T10:52:18.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:18.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:18.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-18T10:52:18.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-18T10:52:18.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-18T10:52:19.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-18T10:52:19.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-18T10:52:19.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-18T10:52:19.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-18T10:52:19.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-18T10:52:19.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-18T10:52:19.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-18T10:52:19.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-18T10:52:19.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-18T10:52:19.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-18T10:52:19.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-18T10:52:19.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-18T10:52:19.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-18T10:52:19.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-18T10:52:19.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-18T10:52:19.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-18T10:52:19.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-18T10:52:19.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-18T10:52:19.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-18T10:52:19.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-18T10:52:19.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-18T10:52:19.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-18T10:52:19.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:19.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:19.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-18T10:52:19.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-18T10:52:20.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-18T10:52:20.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-18T10:52:20.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-18T10:52:20.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-18T10:52:20.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-18T10:52:20.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-18T10:52:20.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-18T10:52:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-18T10:52:20.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-18T10:52:20.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-18T10:52:20.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-18T10:52:20.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-18T10:52:20.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-18T10:52:20.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-18T10:52:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-18T10:52:20.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-18T10:52:20.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-18T10:52:20.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-18T10:52:20.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-18T10:52:20.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-18T10:52:20.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-18T10:52:20.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:20.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:20.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-18T10:52:20.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-18T10:52:21.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-18T10:52:21.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-18T10:52:21.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-18T10:52:21.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-18T10:52:21.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-18T10:52:21.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-18T10:52:21.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-18T10:52:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-18T10:52:21.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-18T10:52:21.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-18T10:52:21.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-18T10:52:21.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-18T10:52:21.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-18T10:52:21.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-18T10:52:21.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-18T10:52:21.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-18T10:52:21.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-18T10:52:21.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-18T10:52:21.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-18T10:52:21.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-18T10:52:21.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-18T10:52:21.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-09-18T10:52:21.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:21.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:21.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-18T10:52:21.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-18T10:52:22.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-18T10:52:22.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-18T10:52:22.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-18T10:52:22.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-18T10:52:22.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-18T10:52:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-18T10:52:22.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-18T10:52:22.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-18T10:52:22.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-18T10:52:22.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-18T10:52:22.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-18T10:52:22.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-18T10:52:22.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-18T10:52:22.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-18T10:52:22.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-18T10:52:22.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-18T10:52:22.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-18T10:52:22.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-18T10:52:22.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-18T10:52:22.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-18T10:52:22.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:22.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:22.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-18T10:52:22.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-09-18T10:52:22.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-18T10:52:23.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-18T10:52:23.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-18T10:52:23.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-18T10:52:23.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-18T10:52:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-18T10:52:23.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-18T10:52:23.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-18T10:52:23.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-18T10:52:23.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-18T10:52:23.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-18T10:52:23.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-18T10:52:23.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-18T10:52:23.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-18T10:52:23.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-18T10:52:23.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-18T10:52:23.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-18T10:52:23.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-18T10:52:23.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-18T10:52:23.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-18T10:52:23.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-18T10:52:23.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-18T10:52:23.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-09-18T10:52:23.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:23.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:23.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-18T10:52:23.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-18T10:52:24.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-18T10:52:24.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-18T10:52:24.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-18T10:52:24.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-18T10:52:24.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-18T10:52:24.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-18T10:52:24.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-18T10:52:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-18T10:52:24.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-18T10:52:24.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-18T10:52:24.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-18T10:52:24.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-18T10:52:24.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-18T10:52:24.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-18T10:52:24.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-18T10:52:24.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-18T10:52:24.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-18T10:52:24.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-18T10:52:24.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-18T10:52:24.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-18T10:52:24.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-09-18T10:52:24.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:24.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:24.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-18T10:52:24.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-18T10:52:25.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-18T10:52:25.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-18T10:52:25.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-18T10:52:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-18T10:52:25.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-18T10:52:25.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-18T10:52:25.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-18T10:52:25.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-18T10:52:25.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-18T10:52:25.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-18T10:52:25.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-18T10:52:25.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-18T10:52:25.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-18T10:52:25.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-18T10:52:25.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-18T10:52:25.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-18T10:52:25.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-18T10:52:25.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-18T10:52:25.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-18T10:52:25.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-18T10:52:25.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-18T10:52:25.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-09-18T10:52:25.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:25.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:25.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-18T10:52:25.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-18T10:52:26.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-18T10:52:26.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-18T10:52:26.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-18T10:52:26.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-18T10:52:26.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-18T10:52:26.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-18T10:52:26.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-18T10:52:26.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-18T10:52:26.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-18T10:52:26.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-18T10:52:26.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-18T10:52:26.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-18T10:52:26.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-18T10:52:26.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-18T10:52:26.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-18T10:52:26.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-18T10:52:26.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-18T10:52:26.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-18T10:52:26.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-18T10:52:26.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-18T10:52:26.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-09-18T10:52:26.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:26.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:26.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-18T10:52:26.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-18T10:52:27.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-18T10:52:27.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-18T10:52:27.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-18T10:52:27.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-18T10:52:27.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-18T10:52:27.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-18T10:52:27.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-18T10:52:27.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-18T10:52:27.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-18T10:52:27.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-18T10:52:27.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-18T10:52:27.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-18T10:52:27.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-18T10:52:27.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-18T10:52:27.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-18T10:52:27.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-18T10:52:27.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-18T10:52:27.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-18T10:52:27.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-18T10:52:27.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-18T10:52:27.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-18T10:52:27.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-09-18T10:52:27.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:27.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:27.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-18T10:52:27.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-18T10:52:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-18T10:52:28.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-18T10:52:28.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-18T10:52:28.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.171 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-09-18T10:52:28.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-18T10:52:28.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-18T10:52:28.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-18T10:52:28.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-18T10:52:28.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-18T10:52:28.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-18T10:52:28.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-18T10:52:28.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-18T10:52:28.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-18T10:52:28.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-18T10:52:28.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-18T10:52:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-18T10:52:28.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-18T10:52:28.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-18T10:52:28.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-18T10:52:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-18T10:52:28.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-18T10:52:28.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-18T10:52:28.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-09-18T10:52:28.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:28.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:28.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-18T10:52:28.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-18T10:52:29.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-18T10:52:29.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-18T10:52:29.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-18T10:52:29.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-18T10:52:29.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-18T10:52:29.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-18T10:52:29.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-18T10:52:29.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-18T10:52:29.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-18T10:52:29.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-18T10:52:29.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-18T10:52:29.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-18T10:52:29.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-18T10:52:29.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-18T10:52:29.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-18T10:52:29.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-18T10:52:29.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-18T10:52:29.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-18T10:52:29.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-18T10:52:29.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-18T10:52:29.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-18T10:52:29.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-18T10:52:29.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-09-18T10:52:29.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:29.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:29.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-18T10:52:29.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-18T10:52:30.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-18T10:52:30.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-18T10:52:30.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-18T10:52:30.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-18T10:52:30.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-18T10:52:30.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-18T10:52:30.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-18T10:52:30.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-09-18T10:52:30.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T10:52:30.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-18T10:52:30.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T10:52:30.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T10:52:31.002 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T10:52:31.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T10:52:31.907 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T10:52:31.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T10:52:31.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T10:52:31.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:52:31.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T10:52:31.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:52:31.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:52:31.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:52:31.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:52:31.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:52:32.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:52:32.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:52:32.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:52:32.239 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:52:32.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:52:32.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:52:32.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:52:32.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:52:32.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:52:32.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:52:32.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:52:32.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:52:32.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:52:32.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T10:52:32.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:52:32.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:32.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:32.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T10:52:32.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:52:32.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:52:32.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:52:32.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:52:32.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:52:32.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:52:32.317 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:52:32.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:52:32.319 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:52:32.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:52:32.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:52:32.321 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:52:32.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:52:32.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:52:32.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:52:32.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:52:32.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:52:32.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:52:32.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T10:52:32.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:52:32.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:32.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:32.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T10:52:32.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:52:32.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:52:32.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T10:52:32.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:52:32.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:32.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:32.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T10:52:32.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:52:32.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:52:32.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:52:32.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T10:52:32.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T10:52:32.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T10:52:32.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:52:32.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:52:32.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:52:32.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T10:52:32.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T10:52:32.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:52:32.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:52:32.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:52:32.709 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:52:32.709 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:32.710 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:32.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:52:32.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T10:52:33.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:52:33.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:52:33.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:52:33.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:52:33.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:52:33.015 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:52:33.016 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:52:33.016 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:52:33.016 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:52:33.017 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:52:33.017 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:33.017 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:33.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:52:33.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:52:33.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T10:52:33.111 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:52:33.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:52:33.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:52:33.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:52:33.113 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:52:33.113 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:52:33.113 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:52:33.113 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:52:33.113 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:52:33.114 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:33.114 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:33.114 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:52:33.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:52:33.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T10:52:33.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T10:52:33.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T10:52:33.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:52:33.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:52:33.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:52:33.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T10:52:33.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:52:34.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:52:34.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:52:34.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:52:34.121 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:52:34.121 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:34.121 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:34.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:52:34.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:52:34.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:52:34.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T10:52:34.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T10:52:34.479 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T10:52:34.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:52:35.495 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T10:52:35.495 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:52:35.495 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:52:35.496 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:52:35.496 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:35.496 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:35.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:52:35.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T10:52:35.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T10:52:35.575 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:52:35.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T10:52:35.585 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:52:32.746+0000 7f5b00b7ca00 0 load: jerasure load: lrc 2024-09-18T10:52:35.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-09-18T10:52:35.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-09-18T10:52:35.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-09-18T10:52:35.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-09-18T10:52:35.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-18T10:52:35.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T10:52:35.587 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-18T10:52:35.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-18T10:52:35.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:52:35.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T10:52:35.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:52:35.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T10:52:35.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:52:35.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:52:35.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:52:35.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:52:35.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:52:35.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:52:35.593 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:52:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:52:35.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:52:35.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:52:35.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:52:35.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T10:52:35.595 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:52:35.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fda60dab-36a1-4e1b-a94e-759962777572 2024-09-18T10:52:35.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fda60dab-36a1-4e1b-a94e-759962777572' 2024-09-18T10:52:35.596 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 fda60dab-36a1-4e1b-a94e-759962777572 2024-09-18T10:52:35.596 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:52:35.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzsOpmmLiNJBAAIVSSgep2QTHLtP6cYt4ZVw== 2024-09-18T10:52:35.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzsOpmmLiNJBAAIVSSgep2QTHLtP6cYt4ZVw=="}' 2024-09-18T10:52:35.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fda60dab-36a1-4e1b-a94e-759962777572 -i td/test-erasure-eio/0/new.json 2024-09-18T10:52:35.959 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:52:35.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T10:52:35.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzsOpmmLiNJBAAIVSSgep2QTHLtP6cYt4ZVw== --osd-uuid fda60dab-36a1-4e1b-a94e-759962777572 2024-09-18T10:52:36.008 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:35.998+0000 7f8dd4a32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:52:36.012 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:36.002+0000 7f8dd4a32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:52:36.015 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:36.006+0000 7f8dd4a32540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:52:36.016 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:36.006+0000 7f8dd4a32540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:52:36.016 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:36.006+0000 7f8dd4a32540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:52:36.016 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:36.006+0000 7f8dd4a32540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:52:36.117 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:36.110+0000 7f8dd4a32540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T10:52:41.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T10:52:41.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:52:41.928 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:52:41.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:52:41.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:52:42.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:52:42.469 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:52:42.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:52:42.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:52:42.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:52:42.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:52:42.491 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:42.482+0000 7f34ac9d4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:52:42.493 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:42.486+0000 7f34ac9d4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:52:42.498 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:42.486+0000 7f34ac9d4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:52:42.901 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:52:42.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:42.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:43.047 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:43.038+0000 7f34ac9d4540 -1 Falling back to public interface 2024-09-18T10:52:43.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:44.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:44.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:44.285 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:52:44.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:52:44.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:44.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:44.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:45.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:45.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:45.685 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:52:45.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:52:45.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:45.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:45.772 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:52:45.762+0000 7f34ac9d4540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:52:46.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:47.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:47.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:47.181 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:52:47.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:52:47.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:47.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:47.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:48.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:48.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:48.668 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:52:48.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:52:48.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:48.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:49.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:50.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:50.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:50.081 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:52:50.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:52:50.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:50.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:50.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:51.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:51.598 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:52:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:52:51.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:51.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:52.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:53.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:53.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:53.056 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:52:53.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:52:53.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:53.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:53.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:54.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:54.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:54.544 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:52:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:52:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:54.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:54.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:55.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:55.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:55.985 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:52:55.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:52:55.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:55.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:56.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:57.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:57.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:57.445 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:52:57.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:52:57.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:57.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:58.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:52:59.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:52:59.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:52:59.206 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:52:59.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:52:59.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:52:59.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:52:59.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:00.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:00.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:00.912 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:53:00.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:53:00.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:00.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:53:01.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:02.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:02.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:02.521 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:53:02.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:53:02.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:02.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:53:03.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:04.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:04.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:04.322 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:53:04.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:53:04.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:04.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:53:04.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:05.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:05.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:05.880 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:53:05.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:53:05.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:05.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:53:06.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:07.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:07.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:07.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:53:07.509 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:53:07.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:07.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:53:08.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:09.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:09.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:09.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:53:09.293 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:53:09.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:09.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:53:09.895 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/105482,v1:127.0.0.1:6803/105482] [v2:127.0.0.1:6804/105482,v1:127.0.0.1:6805/105482] exists,up fda60dab-36a1-4e1b-a94e-759962777572 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T10:53:09.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:53:09.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:53:09.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:53:09.901 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:53:09.901 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:53:09.901 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:53:09.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:53:09.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:53:09.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T10:53:09.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:53:09.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a3649b8-7e51-4fd7-8a8b-ea77e63271a8 2024-09-18T10:53:09.904 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 4a3649b8-7e51-4fd7-8a8b-ea77e63271a8 2024-09-18T10:53:09.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4a3649b8-7e51-4fd7-8a8b-ea77e63271a8' 2024-09-18T10:53:09.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:53:09.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVsepmDjbNNhAA8i2kN4raF1ik3LRtsUrFMg== 2024-09-18T10:53:09.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVsepmDjbNNhAA8i2kN4raF1ik3LRtsUrFMg=="}' 2024-09-18T10:53:09.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a3649b8-7e51-4fd7-8a8b-ea77e63271a8 -i td/test-erasure-eio/1/new.json 2024-09-18T10:53:10.480 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:53:10.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T10:53:10.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVsepmDjbNNhAA8i2kN4raF1ik3LRtsUrFMg== --osd-uuid 4a3649b8-7e51-4fd7-8a8b-ea77e63271a8 2024-09-18T10:53:10.528 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.518+0000 7f77c9071540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:10.530 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.522+0000 7f77c9071540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:10.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.522+0000 7f77c9071540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:10.533 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.526+0000 7f77c9071540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:53:10.533 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.526+0000 7f77c9071540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:53:10.533 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.526+0000 7f77c9071540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:53:10.623 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:10.614+0000 7f77c9071540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T10:53:16.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T10:53:16.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:53:16.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T10:53:16.596 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T10:53:16.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:53:17.132 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T10:53:17.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T10:53:17.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:53:17.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:53:17.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:53:17.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:53:17.163 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:17.154+0000 7fc6598a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:17.165 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:17.158+0000 7fc6598a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:17.166 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:17.158+0000 7fc6598a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:17.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T10:53:17.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:53:17.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:17.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:17.719 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:17.710+0000 7fc6598a3540 -1 Falling back to public interface 2024-09-18T10:53:17.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:18.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:18.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:18.940 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:53:18.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:53:18.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:18.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:19.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:20.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:20.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:20.414 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:53:20.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:53:20.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:20.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:20.493 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:20.486+0000 7fc6598a3540 -1 osd.1 0 log_to_monitors true 2024-09-18T10:53:20.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:21.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:21.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:21.811 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:53:21.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:53:21.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:21.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:22.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:23.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:23.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:23.309 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:53:23.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:53:23.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:23.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:23.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:24.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:24.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:24.768 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:53:24.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:53:24.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:24.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:25.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:26.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:26.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:26.269 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:53:26.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:53:26.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:26.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:26.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:27.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:27.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:27.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:53:27.735 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:53:27.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:27.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:28.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:29.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:29.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:29.319 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:53:29.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:53:29.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:29.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:29.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:30.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:30.731 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:53:30.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:53:30.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:30.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:31.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:32.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:32.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:32.308 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:53:32.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:53:32.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:32.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:32.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:33.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:33.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:33.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:53:33.808 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:53:33.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:33.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:34.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:35.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:35.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:35.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:53:35.558 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:53:35.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:35.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:36.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:37.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:37.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:37.123 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:53:37.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:53:37.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:37.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:38.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:39.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:39.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:39.168 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:53:39.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:53:39.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:39.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:39.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:40.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:40.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:40.732 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:53:40.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:53:40.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:40.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:41.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:42.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:42.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:42.231 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:53:42.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:53:42.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:42.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:42.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:43.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:43.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:43.764 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:53:43.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:53:43.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:43.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:44.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:45.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:45.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:45.379 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:53:45.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:53:45.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:45.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:53:45.844 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/107258,v1:127.0.0.1:6811/107258] [v2:127.0.0.1:6812/107258,v1:127.0.0.1:6813/107258] exists,up 4a3649b8-7e51-4fd7-8a8b-ea77e63271a8 2024-09-18T10:53:45.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:53:45.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:53:45.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:53:45.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T10:53:45.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:53:45.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:53:45.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:53:45.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:53:45.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:53:45.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:53:45.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T10:53:45.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:53:45.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70690253-6033-473e-9e34-e98278221a59 2024-09-18T10:53:45.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 70690253-6033-473e-9e34-e98278221a59' 2024-09-18T10:53:45.853 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 70690253-6033-473e-9e34-e98278221a59 2024-09-18T10:53:45.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:53:45.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5sepm+s7fMxAAnH18pJV5PaNrPEucp1ORHw== 2024-09-18T10:53:45.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5sepm+s7fMxAAnH18pJV5PaNrPEucp1ORHw=="}' 2024-09-18T10:53:45.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70690253-6033-473e-9e34-e98278221a59 -i td/test-erasure-eio/2/new.json 2024-09-18T10:53:46.444 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:53:46.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T10:53:46.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5sepm+s7fMxAAnH18pJV5PaNrPEucp1ORHw== --osd-uuid 70690253-6033-473e-9e34-e98278221a59 2024-09-18T10:53:46.497 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.486+0000 7f1fe8c1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:46.525 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.518+0000 7f1fe8c1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:46.526 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.518+0000 7f1fe8c1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:46.527 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.518+0000 7f1fe8c1d540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:53:46.527 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.518+0000 7f1fe8c1d540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:53:46.527 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.518+0000 7f1fe8c1d540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:53:46.620 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:46.610+0000 7f1fe8c1d540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T10:53:53.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T10:53:53.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:53:53.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T10:53:53.020 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T10:53:53.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:53:53.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T10:53:53.598 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:53:53.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:53:53.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:53:53.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:53:53.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:53:53.628 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:53.617+0000 7f469a2c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:53.630 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:53.621+0000 7f469a2c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:53.632 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:53.621+0000 7f469a2c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:53:53.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T10:53:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:53:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:53:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:53:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:53:53.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:53.995 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:53:53.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:53:53.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:53.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:53:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:54.177+0000 7f469a2c7540 -1 Falling back to public interface 2024-09-18T10:53:54.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:55.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:55.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:55.499 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:53:55.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:53:55.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:55.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:53:55.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:56.851 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:53:56.841+0000 7f469a2c7540 -1 osd.2 0 log_to_monitors true 2024-09-18T10:53:56.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:56.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:56.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:53:56.946 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:53:56.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:56.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:53:57.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:53:58.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:53:58.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:53:58.633 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:53:58.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:53:58.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:53:58.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:53:59.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:00.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:00.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:00.132 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:54:00.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:54:00.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:00.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:00.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:01.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:01.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:01.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:54:01.670 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:54:01.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:01.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:02.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:03.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:03.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:03.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:54:03.136 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:54:03.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:03.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:03.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:04.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:04.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:04.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:54:04.569 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:54:04.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:04.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:04.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:05.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:05.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:05.998 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:54:05.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:54:05.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:05.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:06.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:07.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:07.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:07.532 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:54:07.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:54:07.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:07.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:08.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:09.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:09.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:09.106 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:54:09.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:54:09.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:09.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:09.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:10.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:10.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:10.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:54:10.593 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:54:10.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:10.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:11.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:12.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:12.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:12.713 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:54:12.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:54:12.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:12.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:13.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:14.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:14.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:14.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:54:14.296 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:54:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:14.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:14.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:15.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:15.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:15.975 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:54:15.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:54:15.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:15.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:16.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:17.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:17.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:17.595 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:54:17.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:54:17.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:17.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:18.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:19.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:19.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:19.122 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:54:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:54:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:19.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:20.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:20.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:20.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:54:20.926 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:54:20.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:20.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:54:21.629 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/109059,v1:127.0.0.1:6819/109059] [v2:127.0.0.1:6820/109059,v1:127.0.0.1:6821/109059] exists,up 70690253-6033-473e-9e34-e98278221a59 2024-09-18T10:54:21.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:54:21.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T10:54:21.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:54:21.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:54:21.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:54:21.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:54:21.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:54:21.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=39dc2404-a9f0-400e-9423-6dbb22e2953b 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 39dc2404-a9f0-400e-9423-6dbb22e2953b' 2024-09-18T10:54:21.638 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 39dc2404-a9f0-400e-9423-6dbb22e2953b 2024-09-18T10:54:21.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:54:21.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdsepmkdgDJxAA6/4y1zAvU+OhtD3Or1zNxg== 2024-09-18T10:54:21.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdsepmkdgDJxAA6/4y1zAvU+OhtD3Or1zNxg=="}' 2024-09-18T10:54:21.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 39dc2404-a9f0-400e-9423-6dbb22e2953b -i td/test-erasure-eio/3/new.json 2024-09-18T10:54:22.663 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:54:22.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T10:54:22.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdsepmkdgDJxAA6/4y1zAvU+OhtD3Or1zNxg== --osd-uuid 39dc2404-a9f0-400e-9423-6dbb22e2953b 2024-09-18T10:54:22.708 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.697+0000 7fb2d885f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:54:22.789 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.781+0000 7fb2d885f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:54:22.791 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.781+0000 7fb2d885f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:54:22.792 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.781+0000 7fb2d885f540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:54:22.792 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.781+0000 7fb2d885f540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:54:22.792 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.781+0000 7fb2d885f540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:54:22.859 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:22.853+0000 7fb2d885f540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T10:54:29.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T10:54:29.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:54:29.147 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T10:54:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T10:54:29.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:54:29.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T10:54:29.682 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T10:54:29.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:54:29.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:54:29.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:54:29.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:54:29.711 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:29.701+0000 7fd50c355540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:54:29.714 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:29.705+0000 7fd50c355540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:54:29.716 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:29.705+0000 7fd50c355540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:54:30.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T10:54:30.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:54:30.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T10:54:30.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:54:30.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:54:30.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:30.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:54:30.152 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:54:30.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:30.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:30.287 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:30.277+0000 7fd50c355540 -1 Falling back to public interface 2024-09-18T10:54:30.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:31.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:31.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:31.566 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:54:31.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:54:31.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:31.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:31.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:32.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:32.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:32.978 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:54:32.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:54:32.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:32.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:33.008 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:54:32.997+0000 7fd50c355540 -1 osd.3 0 log_to_monitors true 2024-09-18T10:54:33.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:34.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:34.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:34.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:54:34.509 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:54:34.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:34.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:35.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:36.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:36.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:36.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:54:36.017 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:54:36.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:36.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:36.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:37.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:37.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:37.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:54:37.717 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:54:37.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:37.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:38.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:39.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:39.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:39.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:54:39.133 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:54:39.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:39.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:39.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:40.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:40.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:40.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:54:40.558 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:54:40.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:40.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:40.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:41.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:41.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:41.973 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:54:41.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:54:41.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:41.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:42.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:43.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:43.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:43.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:54:43.540 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:54:43.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:43.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:43.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:44.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:44.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:44.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:54:44.980 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:54:44.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:44.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:45.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:46.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:46.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:46.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:54:46.413 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:54:46.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:46.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:47.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:48.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:48.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:48.137 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:54:48.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:54:48.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:48.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:48.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:49.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:49.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:49.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:54:49.835 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:54:49.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:49.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:50.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:51.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:51.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:51.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:54:51.267 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:54:51.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:51.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:52.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:53.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:53.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:53.270 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:54:53.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:54:53.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:53.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:53.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:54.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:54.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:54.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:54:54.895 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:54:54.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:54.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:55.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:56.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:56.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:56.517 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:54:56.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:54:56.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:56.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:57.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:54:58.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:54:58.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:54:58.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:54:58.559 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:54:58.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:54:58.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:54:59.163 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/110834,v1:127.0.0.1:6827/110834] [v2:127.0.0.1:6828/110834,v1:127.0.0.1:6829/110834] exists,up 39dc2404-a9f0-400e-9423-6dbb22e2953b 2024-09-18T10:54:59.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:54:59.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:54:59.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:54:59.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:54:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:54:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:54:59.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:54:59.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:54:59.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T10:54:59.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:54:59.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86bbd3a2-d158-4d40-94be-05e67024b1fe 2024-09-18T10:54:59.171 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 86bbd3a2-d158-4d40-94be-05e67024b1fe 2024-09-18T10:54:59.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 86bbd3a2-d158-4d40-94be-05e67024b1fe' 2024-09-18T10:54:59.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:54:59.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDsepmMfQuCxAAripY0y8YZPFEFRmFgCR1Rg== 2024-09-18T10:54:59.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDsepmMfQuCxAAripY0y8YZPFEFRmFgCR1Rg=="}' 2024-09-18T10:54:59.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86bbd3a2-d158-4d40-94be-05e67024b1fe -i td/test-erasure-eio/4/new.json 2024-09-18T10:55:00.226 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:55:00.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T10:55:00.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDsepmMfQuCxAAripY0y8YZPFEFRmFgCR1Rg== --osd-uuid 86bbd3a2-d158-4d40-94be-05e67024b1fe 2024-09-18T10:55:00.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.265+0000 7f9ee1b6c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:00.276 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.269+0000 7f9ee1b6c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:00.278 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.273+0000 7f9ee1b6c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:00.279 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.273+0000 7f9ee1b6c540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:55:00.279 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.273+0000 7f9ee1b6c540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:55:00.279 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.273+0000 7f9ee1b6c540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:55:00.365 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:00.361+0000 7f9ee1b6c540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T10:55:06.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T10:55:06.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:55:06.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T10:55:06.567 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T10:55:06.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:55:07.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T10:55:07.130 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T10:55:07.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:55:07.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:55:07.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:55:07.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:55:07.159 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:07.153+0000 7f6ec153e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:07.161 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:07.157+0000 7f6ec153e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:07.162 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:07.157+0000 7f6ec153e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:07.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:55:07.535 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:55:07.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:07.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:07.699 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:07.693+0000 7f6ec153e540 -1 Falling back to public interface 2024-09-18T10:55:07.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:08.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:08.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:08.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:55:08.921 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:55:08.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:08.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:09.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:10.201+0000 7f6ec153e540 -1 osd.4 0 log_to_monitors true 2024-09-18T10:55:10.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:10.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:10.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:55:10.312 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:55:10.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:10.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:11.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:12.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:12.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:12.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:55:12.004 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:55:12.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:12.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:13.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:14.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:14.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:14.023 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:55:14.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:55:14.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:14.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:14.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:15.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:15.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:15.529 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:55:15.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:55:15.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:15.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:16.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:17.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:17.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:17.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:55:17.050 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:55:17.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:17.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:17.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:18.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:18.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:18.539 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:55:18.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:55:18.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:18.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:19.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:20.036 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:55:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:55:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:20.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:21.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:21.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:21.588 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:55:21.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:55:21.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:21.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:22.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:23.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:23.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:23.125 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:55:23.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:55:23.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:23.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:23.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:24.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:24.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:24.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:55:24.570 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:55:24.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:24.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:25.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:26.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:26.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:26.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:55:26.190 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:55:26.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:26.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:26.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:27.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:27.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:27.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:55:27.748 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:55:27.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:27.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:28.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:29.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:29.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:29.442 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:55:29.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:55:29.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:29.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:30.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:31.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:31.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:31.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:55:31.269 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:55:31.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:31.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:32.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:33.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:33.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:33.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:55:33.285 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:55:33.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:33.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:33.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:34.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:34.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:34.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:55:34.722 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:55:34.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:34.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:35.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:36.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:36.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:36.314 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:55:36.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:55:36.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:36.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:55:37.063 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/112640,v1:127.0.0.1:6835/112640] [v2:127.0.0.1:6836/112640,v1:127.0.0.1:6837/112640] exists,up 86bbd3a2-d158-4d40-94be-05e67024b1fe 2024-09-18T10:55:37.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:55:37.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:55:37.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:55:37.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:55:37.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-18T10:55:37.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:55:37.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:55:37.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T10:55:37.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:55:37.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-18T10:55:37.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:55:37.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:55:37.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:55:37.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:55:37.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-18T10:55:37.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-18T10:55:37.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:55:37.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:55:37.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:55:37.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:55:37.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:55:37.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:55:37.075 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:55:37.075 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:55:37.076 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:55:37.076 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:55:37.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:55:37.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:55:37.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:55:37.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:55:37.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:55:37.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:55:37.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:55:37.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:55:37.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:55:37.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:55:37.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:55:37.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:55:37.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-18T10:55:37.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:55:37.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14d34380-7585-45f5-8e9c-262835ff90d0 2024-09-18T10:55:37.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 14d34380-7585-45f5-8e9c-262835ff90d0' 2024-09-18T10:55:37.081 INFO:tasks.workunit.client.0.smithi174.stdout:add osd5 14d34380-7585-45f5-8e9c-262835ff90d0 2024-09-18T10:55:37.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:55:37.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpsepmXeZSBRAAtDgxwTL97Z0lFz1+z6TRow== 2024-09-18T10:55:37.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpsepmXeZSBRAAtDgxwTL97Z0lFz1+z6TRow=="}' 2024-09-18T10:55:37.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14d34380-7585-45f5-8e9c-262835ff90d0 -i td/test-erasure-eio/5/new.json 2024-09-18T10:55:37.885 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:55:37.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-18T10:55:37.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpsepmXeZSBRAAtDgxwTL97Z0lFz1+z6TRow== --osd-uuid 14d34380-7585-45f5-8e9c-262835ff90d0 2024-09-18T10:55:37.932 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:37.924+0000 7f47e8d72540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:38.088 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:38.084+0000 7f47e8d72540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:38.091 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:38.084+0000 7f47e8d72540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:38.092 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:38.084+0000 7f47e8d72540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T10:55:38.092 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:38.084+0000 7f47e8d72540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T10:55:38.092 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:38.084+0000 7f47e8d72540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T10:55:38.227 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:38.220+0000 7f47e8d72540 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-18T10:55:44.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-18T10:55:44.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:55:44.859 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd5 key to auth repository 2024-09-18T10:55:44.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T10:55:44.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:55:45.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T10:55:45.453 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.5 2024-09-18T10:55:45.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:55:45.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:55:45.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:55:45.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:55:45.482 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:45.476+0000 7ff68d55c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:45.486 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:45.480+0000 7ff68d55c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:45.491 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:45.480+0000 7ff68d55c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:55:45.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:55:45.907 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:55:45.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:45.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:46.051 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:46.044+0000 7ff68d55c540 -1 Falling back to public interface 2024-09-18T10:55:46.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:47.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:47.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:47.359 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:55:47.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:55:47.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:47.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:47.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:48.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:48.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:48.769 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:55:48.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:55:48.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:48.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:48.848 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:55:48.840+0000 7ff68d55c540 -1 osd.5 0 log_to_monitors true 2024-09-18T10:55:49.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:50.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:50.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:50.218 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:55:50.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:55:50.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:50.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:50.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:51.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:51.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:51.902 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:55:51.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:55:51.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:51.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:52.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:53.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:53.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:53.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:55:53.348 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:55:53.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:53.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:53.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:54.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:54.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:54.890 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:55:54.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:55:54.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:54.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:55.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:56.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:56.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:56.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:55:56.350 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:55:56.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:56.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:56.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:57.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:57.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:57.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:55:57.958 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:55:57.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:57.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:55:58.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:55:59.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:55:59.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:55:59.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:55:59.583 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:55:59.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:55:59.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:00.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:01.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:01.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:01.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:56:01.183 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:56:01.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:01.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:01.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:02.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:02.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:02.804 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:56:02.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:56:02.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:02.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:03.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:04.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:04.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:04.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:56:04.428 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:56:04.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:04.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:05.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:06.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:06.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:06.292 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:56:06.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:56:06.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:06.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:06.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:07.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:07.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:07.758 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:56:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:56:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:08.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:09.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:09.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:09.264 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:56:09.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:56:09.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:09.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:10.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:11.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:11.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:11.278 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:56:11.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:56:11.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:11.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:11.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:12.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:12.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:12.911 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:56:12.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:56:12.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:12.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:13.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:14.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:14.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:14.419 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:56:14.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:56:14.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:14.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T10:56:14.947 INFO:tasks.workunit.client.0.smithi174.stdout:osd.5 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/114441,v1:127.0.0.1:6843/114441] [v2:127.0.0.1:6844/114441,v1:127.0.0.1:6845/114441] exists,up 14d34380-7585-45f5-8e9c-262835ff90d0 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-18T10:56:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:56:14.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:56:14.951 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:56:14.951 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:56:14.951 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:56:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:56:14.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:56:14.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-18T10:56:14.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:56:14.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2c840f0-43a1-48bf-a5f1-ea353b4245f4 2024-09-18T10:56:14.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 e2c840f0-43a1-48bf-a5f1-ea353b4245f4' 2024-09-18T10:56:14.954 INFO:tasks.workunit.client.0.smithi174.stdout:add osd6 e2c840f0-43a1-48bf-a5f1-ea353b4245f4 2024-09-18T10:56:14.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:56:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOsepmO9OZORAAABoHcz1yF3v0/z1cuC7NRQ== 2024-09-18T10:56:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOsepmO9OZORAAABoHcz1yF3v0/z1cuC7NRQ=="}' 2024-09-18T10:56:14.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2c840f0-43a1-48bf-a5f1-ea353b4245f4 -i td/test-erasure-eio/6/new.json 2024-09-18T10:56:15.873 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:56:15.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-18T10:56:15.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOsepmO9OZORAAABoHcz1yF3v0/z1cuC7NRQ== --osd-uuid e2c840f0-43a1-48bf-a5f1-ea353b4245f4 2024-09-18T10:56:15.917 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:15.912+0000 7fe57c278540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:56:15.919 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:15.912+0000 7fe57c278540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:56:15.921 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:15.916+0000 7fe57c278540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:56:15.921 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:15.916+0000 7fe57c278540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T10:56:15.921 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:15.916+0000 7fe57c278540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T10:56:15.921 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:15.916+0000 7fe57c278540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T10:56:16.068 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:16.064+0000 7fe57c278540 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-18T10:56:22.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-18T10:56:22.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:56:22.662 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd6 key to auth repository 2024-09-18T10:56:22.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T10:56:22.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:56:23.227 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.6 2024-09-18T10:56:23.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T10:56:23.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:56:23.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:56:23.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:56:23.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:56:23.256 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:23.252+0000 7fea7d2b7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:56:23.258 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:23.252+0000 7fea7d2b7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:56:23.260 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:23.256+0000 7fea7d2b7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:56:23.692 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:56:23.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:23.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:23.818 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:23.812+0000 7fea7d2b7540 -1 Falling back to public interface 2024-09-18T10:56:24.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:25.137 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:56:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:56:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:25.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:26.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:26.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:26.585 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:56:26.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:56:26.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:26.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:26.695 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:56:26.688+0000 7fea7d2b7540 -1 osd.6 0 log_to_monitors true 2024-09-18T10:56:27.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:28.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:28.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:28.056 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:56:28.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:56:28.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:28.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:28.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:29.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:29.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:29.910 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:56:29.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:56:29.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:29.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:30.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:31.546 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:56:31.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:56:31.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:31.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:31.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:32.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:32.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:32.976 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:56:32.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:56:32.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:32.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:33.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:34.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:34.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:34.385 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:56:34.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:56:34.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:34.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:34.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:35.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:35.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:35.838 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:56:35.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:56:35.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:35.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:36.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:37.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:37.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:37.271 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:56:37.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:56:37.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:37.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:37.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:38.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:38.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:38.779 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:56:38.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:56:38.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:38.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:39.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:40.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:40.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:40.194 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:56:40.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:56:40.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:40.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:41.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:42.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:42.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:42.137 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:56:42.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:56:42.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:42.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:42.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:43.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:43.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:43.710 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:56:43.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:56:43.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:43.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:44.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:45.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:45.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:45.351 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:56:45.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:56:45.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:45.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:46.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:47.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:47.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:47.253 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:56:47.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:56:47.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:47.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:48.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:49.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:49.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:49.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:56:49.100 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:56:49.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:49.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:49.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:50.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:50.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:50.915 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:56:50.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:56:50.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:50.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:51.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:56:52.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:56:52.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:56:52.513 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T10:56:52.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T10:56:52.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:56:52.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T10:56:53.019 INFO:tasks.workunit.client.0.smithi174.stdout:osd.6 up in weight 1 up_from 62 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/116242,v1:127.0.0.1:6851/116242] [v2:127.0.0.1:6852/116242,v1:127.0.0.1:6853/116242] exists,up e2c840f0-43a1-48bf-a5f1-ea353b4245f4 2024-09-18T10:56:53.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:56:53.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:56:53.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:56:53.020 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T10:56:53.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T10:56:53.101 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:56:53.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T10:56:53.120 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:52:43.046+0000 7f34ac9d4540 0 load: jerasure load: lrc 2024-09-18T10:56:53.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-09-18T10:56:53.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-18T10:56:53.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T10:56:53.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T10:56:53.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-18T10:56:53.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T10:56:53.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-18T10:56:53.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T10:56:53.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-18T10:56:53.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T10:56:53.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T10:56:55.028 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T10:56:55.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:56:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T10:56:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:56:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:56:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:56:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:56:56.042 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:56:56.042 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:56:56.043 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:56:56.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:56:56.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:56:56.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:56:56.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:56:56.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:56:56.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:56:56.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:56:56.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:56:56.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:56:56.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:56.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:56:56.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803828 2024-09-18T10:56:56.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803828 2024-09-18T10:56:56.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828' 2024-09-18T10:56:56.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:56.811 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:56:56.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542192 2024-09-18T10:56:56.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542192 2024-09-18T10:56:56.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-60129542192' 2024-09-18T10:56:56.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:56.956 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:56:57.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345968 2024-09-18T10:56:57.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345968 2024-09-18T10:56:57.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-60129542192 2-85899345968' 2024-09-18T10:56:57.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:57.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:56:57.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=133143986213 2024-09-18T10:56:57.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 133143986213 2024-09-18T10:56:57.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-60129542192 2-85899345968 3-133143986213' 2024-09-18T10:56:57.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:57.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:56:57.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626465 2024-09-18T10:56:57.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626465 2024-09-18T10:56:57.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-60129542192 2-85899345968 3-133143986213 4-180388626465' 2024-09-18T10:56:57.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:57.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:56:57.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=223338299413 2024-09-18T10:56:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 223338299413 2024-09-18T10:56:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-60129542192 2-85899345968 3-133143986213 4-180388626465 5-223338299413' 2024-09-18T10:56:57.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:56:57.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:56:57.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=266287972368 2024-09-18T10:56:57.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 266287972368 2024-09-18T10:56:57.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803828 1-60129542192 2-85899345968 3-133143986213 4-180388626465 5-223338299413 6-266287972368' 2024-09-18T10:56:57.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:56:57.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803828 2024-09-18T10:56:57.697 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:56:57.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:56:57.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803828 2024-09-18T10:56:57.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:56:57.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803828 2024-09-18T10:56:57.702 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803828 2024-09-18T10:56:57.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-09-18T10:56:57.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:56:58.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803828 -lt 25769803828 2024-09-18T10:56:58.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:56:58.196 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542192 2024-09-18T10:56:58.197 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:56:58.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:56:58.200 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542192 2024-09-18T10:56:58.200 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:56:58.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542192 2024-09-18T10:56:58.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542192' 2024-09-18T10:56:58.202 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 60129542192 2024-09-18T10:56:58.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:56:58.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542192 -lt 60129542192 2024-09-18T10:56:58.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:56:58.654 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345968 2024-09-18T10:56:58.654 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:56:58.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:56:58.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345968 2024-09-18T10:56:58.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:56:58.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345968 2024-09-18T10:56:58.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345968' 2024-09-18T10:56:58.660 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 85899345968 2024-09-18T10:56:58.660 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:56:59.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345968 -lt 85899345968 2024-09-18T10:56:59.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:56:59.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-133143986213 2024-09-18T10:56:59.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:56:59.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:56:59.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-133143986213 2024-09-18T10:56:59.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:56:59.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=133143986213 2024-09-18T10:56:59.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 133143986213' 2024-09-18T10:56:59.151 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 133143986213 2024-09-18T10:56:59.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:56:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 133143986214 -lt 133143986213 2024-09-18T10:56:59.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:56:59.598 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-180388626465 2024-09-18T10:56:59.598 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:56:59.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:56:59.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-180388626465 2024-09-18T10:56:59.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:56:59.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626465 2024-09-18T10:56:59.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 180388626465' 2024-09-18T10:56:59.603 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 180388626465 2024-09-18T10:56:59.604 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:57:00.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626465 -lt 180388626465 2024-09-18T10:57:00.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:00.045 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-223338299413 2024-09-18T10:57:00.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:00.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:57:00.048 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-223338299413 2024-09-18T10:57:00.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:00.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=223338299413 2024-09-18T10:57:00.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 223338299413' 2024-09-18T10:57:00.051 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 223338299413 2024-09-18T10:57:00.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:57:00.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 223338299413 -lt 223338299413 2024-09-18T10:57:00.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:00.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-266287972368 2024-09-18T10:57:00.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:00.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:57:00.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-266287972368 2024-09-18T10:57:00.533 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:00.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=266287972368 2024-09-18T10:57:00.535 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 266287972368 2024-09-18T10:57:00.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 266287972368' 2024-09-18T10:57:00.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:57:00.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 266287972368 -lt 266287972368 2024-09-18T10:57:00.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:57:00.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:57:00.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:57:01.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T10:57:01.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:57:01.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:57:01.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:57:01.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:57:01.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:57:01.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:57:01.569 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:57:02.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T10:57:02.006 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:57:02.007 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:57:02.007 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:57:02.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T10:57:02.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:57:02.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:57:02.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-09-18T10:57:02.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-09-18T10:57:02.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-18T10:57:02.534 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-09-18T10:57:02.535 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-18T10:57:03.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-09-18T10:57:03.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-18T10:57:03.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:57:03.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:57:03.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:57:03.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:57:03.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:57:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:57:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-18T10:57:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T10:57:03.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T10:57:03.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=mdata 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T10:57:04.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T10:57:04.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T10:57:04.463 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T10:57:04.463 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T10:57:04.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=5 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T10:57:04.882 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/5/type 2024-09-18T10:57:04.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T10:57:04.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-09-18T10:57:04.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T10:57:04.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-09-18T10:57:04.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T10:57:04.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T10:57:04.885 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T10:57:04.885 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-09-18T10:57:04.885 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T10:57:04.885 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T10:57:04.886 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:57:04.886 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:04.886 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:04.886 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.5.asok 2024-09-18T10:57:04.886 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-09-18T10:57:04.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T10:57:04.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T10:57:04.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T10:57:04.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.5 2024-09-18T10:57:04.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-09-18T10:57:04.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-09-18T10:57:04.981 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:57:04.981 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:04.981 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:04.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.5.asok 2024-09-18T10:57:04.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T10:57:04.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-09-18T10:57:05.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-09-18T10:57:05.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T10:57:05.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T10:57:05.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:57:05.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:05.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T10:57:05.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T10:57:05.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T10:57:05.082 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T10:57:05.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:57:05.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-18T10:57:05.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T10:57:05.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T10:57:05.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T10:57:05.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T10:57:05.105 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:05.100+0000 7fd693f80540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:05.107 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:05.100+0000 7fd693f80540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:05.111 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:05.104+0000 7fd693f80540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T10:57:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:57:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T10:57:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:57:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:57:05.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:57:05.554 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:57:05.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:05.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:57:05.645 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:05.640+0000 7fd693f80540 -1 Falling back to public interface 2024-09-18T10:57:06.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:07.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:07.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:07.007 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:57:07.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:57:07.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:07.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:57:07.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:08.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:08.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:08.430 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:57:08.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:57:08.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:08.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:57:08.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:08.896 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:08.892+0000 7fd693f80540 -1 osd.3 70 log_to_monitors true 2024-09-18T10:57:09.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:09.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:09.842 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:57:09.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:57:09.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:09.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:57:10.336 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [31,70) [v2:127.0.0.1:6826/117951,v1:127.0.0.1:6827/117951] [v2:127.0.0.1:6828/117951,v1:127.0.0.1:6829/117951] exists,up 39dc2404-a9f0-400e-9423-6dbb22e2953b 2024-09-18T10:57:10.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:57:10.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:57:10.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:57:10.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-09-18T10:57:10.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T10:57:10.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T10:57:10.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T10:57:10.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T10:57:10.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T10:57:10.341 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T10:57:10.341 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T10:57:10.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T10:57:10.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T10:57:10.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T10:57:10.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T10:57:10.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T10:57:10.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T10:57:10.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T10:57:10.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T10:57:11.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T10:57:11.032 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T10:57:11.032 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T10:57:11.032 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T10:57:11.032 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T10:57:11.033 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T10:57:11.033 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T10:57:11.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T10:57:11.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T10:57:11.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803833 2024-09-18T10:57:11.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803833 2024-09-18T10:57:11.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833' 2024-09-18T10:57:11.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T10:57:11.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542197 2024-09-18T10:57:11.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542197 2024-09-18T10:57:11.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542197' 2024-09-18T10:57:11.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T10:57:11.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345973 2024-09-18T10:57:11.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345973 2024-09-18T10:57:11.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542197 2-85899345973' 2024-09-18T10:57:11.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.531 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T10:57:11.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=317827579907 2024-09-18T10:57:11.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 317827579907 2024-09-18T10:57:11.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542197 2-85899345973 3-317827579907' 2024-09-18T10:57:11.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T10:57:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626470 2024-09-18T10:57:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626470 2024-09-18T10:57:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542197 2-85899345973 3-317827579907 4-180388626470' 2024-09-18T10:57:11.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.822 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T10:57:11.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=223338299418 2024-09-18T10:57:11.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 223338299418 2024-09-18T10:57:11.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542197 2-85899345973 3-317827579907 4-180388626470 5-223338299418' 2024-09-18T10:57:11.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T10:57:11.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T10:57:12.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=266287972372 2024-09-18T10:57:12.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 266287972372 2024-09-18T10:57:12.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542197 2-85899345973 3-317827579907 4-180388626470 5-223338299418 6-266287972372' 2024-09-18T10:57:12.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:12.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803833 2024-09-18T10:57:12.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:12.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T10:57:12.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803833 2024-09-18T10:57:12.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:12.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803833 2024-09-18T10:57:12.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-09-18T10:57:12.126 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803833 2024-09-18T10:57:12.126 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T10:57:12.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803833 -lt 25769803833 2024-09-18T10:57:12.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:12.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542197 2024-09-18T10:57:12.533 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:12.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T10:57:12.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542197 2024-09-18T10:57:12.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:12.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542197 2024-09-18T10:57:12.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542197' 2024-09-18T10:57:12.538 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 60129542197 2024-09-18T10:57:12.539 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:57:13.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542195 -lt 60129542197 2024-09-18T10:57:13.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T10:57:14.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T10:57:14.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T10:57:14.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542197 -lt 60129542197 2024-09-18T10:57:14.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:14.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345973 2024-09-18T10:57:14.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:14.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T10:57:14.485 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345973 2024-09-18T10:57:14.485 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:14.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345973 2024-09-18T10:57:14.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345973' 2024-09-18T10:57:14.487 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 85899345973 2024-09-18T10:57:14.488 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T10:57:14.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345973 -lt 85899345973 2024-09-18T10:57:14.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:14.923 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:14.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-317827579907 2024-09-18T10:57:14.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T10:57:14.926 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-317827579907 2024-09-18T10:57:14.926 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:14.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=317827579907 2024-09-18T10:57:14.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 317827579907' 2024-09-18T10:57:14.929 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 317827579907 2024-09-18T10:57:14.929 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T10:57:15.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 317827579907 -lt 317827579907 2024-09-18T10:57:15.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:15.347 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-180388626470 2024-09-18T10:57:15.347 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:15.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T10:57:15.350 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-180388626470 2024-09-18T10:57:15.350 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:15.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626470 2024-09-18T10:57:15.353 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 180388626470 2024-09-18T10:57:15.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 180388626470' 2024-09-18T10:57:15.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T10:57:15.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626470 -lt 180388626470 2024-09-18T10:57:15.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:15.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-223338299418 2024-09-18T10:57:15.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:15.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T10:57:15.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-223338299418 2024-09-18T10:57:15.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:15.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=223338299418 2024-09-18T10:57:15.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 223338299418' 2024-09-18T10:57:15.791 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 223338299418 2024-09-18T10:57:15.791 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T10:57:16.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 223338299419 -lt 223338299418 2024-09-18T10:57:16.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T10:57:16.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-266287972372 2024-09-18T10:57:16.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T10:57:16.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T10:57:16.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T10:57:16.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-266287972372 2024-09-18T10:57:16.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=266287972372 2024-09-18T10:57:16.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 266287972372' 2024-09-18T10:57:16.187 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 266287972372 2024-09-18T10:57:16.187 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T10:57:16.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 266287972373 -lt 266287972372 2024-09-18T10:57:16.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T10:57:16.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:57:16.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:57:17.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T10:57:17.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T10:57:17.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T10:57:17.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T10:57:17.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T10:57:17.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T10:57:17.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T10:57:17.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T10:57:17.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T10:57:17.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T10:57:17.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T10:57:17.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T10:57:18.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T10:57:18.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-18T10:57:18.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T10:57:18.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T10:57:18.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-09-18T10:57:18.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T10:57:18.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T10:57:18.668 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T10:57:18.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T10:57:19.305 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T10:57:19.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T10:57:19.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T10:57:19.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:57:19.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T10:57:19.315 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:57:19.315 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:57:19.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:57:19.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:57:19.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:57:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:57:19.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:57:19.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:57:19.438 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:57:19.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:57:19.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:57:19.440 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:57:19.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:57:19.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:57:19.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:57:19.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:57:19.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:57:19.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:57:19.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T10:57:19.476 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:57:19.476 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:19.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:19.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T10:57:19.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T10:57:19.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T10:57:19.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T10:57:19.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T10:57:19.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T10:57:19.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T10:57:19.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T10:57:19.485 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T10:57:19.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T10:57:19.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T10:57:19.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T10:57:19.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T10:57:19.488 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T10:57:19.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:57:19.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T10:57:19.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T10:57:19.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T10:57:19.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T10:57:19.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T10:57:19.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T10:57:19.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T10:57:19.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:19.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:19.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T10:57:19.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T10:57:19.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T10:57:19.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T10:57:19.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T10:57:19.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:19.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:19.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T10:57:19.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T10:57:19.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T10:57:19.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T10:57:19.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T10:57:19.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T10:57:19.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T10:57:19.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:57:19.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:57:19.910 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:57:19.910 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:19.910 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:19.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:19.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T10:57:20.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T10:57:20.192 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T10:57:20.192 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:57:20.192 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:57:20.192 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T10:57:20.193 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T10:57:20.193 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:57:20.193 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:57:20.193 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:57:20.194 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:57:20.194 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:20.194 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:20.194 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:57:20.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:57:20.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T10:57:20.290 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T10:57:20.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T10:57:20.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T10:57:20.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T10:57:20.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T10:57:20.292 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T10:57:20.292 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:57:20.292 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:57:20.292 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:57:20.293 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:20.293 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:20.293 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:57:20.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T10:57:20.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T10:57:20.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T10:57:21.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T10:57:21.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:57:21.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:57:21.279 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:57:21.279 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:21.279 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:21.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:21.280 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:57:21.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T10:57:21.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T10:57:21.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T10:57:21.661 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T10:57:21.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T10:57:22.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T10:57:22.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T10:57:22.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T10:57:22.673 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T10:57:22.673 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:22.673 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:22.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T10:57:22.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T10:57:22.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T10:57:22.750 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T10:57:22.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T10:57:22.761 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:57:19.952+0000 7fe44edcfa00 0 load: jerasure load: lrc 2024-09-18T10:57:22.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-09-18T10:57:22.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-09-18T10:57:22.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-09-18T10:57:22.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-09-18T10:57:22.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-09-18T10:57:22.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T10:57:22.763 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-09-18T10:57:22.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-09-18T10:57:22.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:57:22.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T10:57:22.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:57:22.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:57:22.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:57:22.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:57:22.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:57:22.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:57:22.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:57:22.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:57:22.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:57:22.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T10:57:22.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:57:22.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=512c7035-e619-49d8-9576-574dba518018 2024-09-18T10:57:22.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 512c7035-e619-49d8-9576-574dba518018' 2024-09-18T10:57:22.773 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 512c7035-e619-49d8-9576-574dba518018 2024-09-18T10:57:22.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:57:22.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQASsupmbB0QLxAAigYituTxwW9LI1yR06LcaQ== 2024-09-18T10:57:22.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQASsupmbB0QLxAAigYituTxwW9LI1yR06LcaQ=="}' 2024-09-18T10:57:22.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 512c7035-e619-49d8-9576-574dba518018 -i td/test-erasure-eio/0/new.json 2024-09-18T10:57:23.172 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:57:23.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T10:57:23.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASsupmbB0QLxAAigYituTxwW9LI1yR06LcaQ== --osd-uuid 512c7035-e619-49d8-9576-574dba518018 2024-09-18T10:57:23.219 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.216+0000 7fcbd06c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:23.222 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.216+0000 7fcbd06c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:23.223 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.220+0000 7fcbd06c7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:23.223 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.220+0000 7fcbd06c7540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:57:23.224 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.220+0000 7fcbd06c7540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:57:23.224 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.220+0000 7fcbd06c7540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T10:57:23.304 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:23.300+0000 7fcbd06c7540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T10:57:29.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T10:57:29.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:57:29.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T10:57:29.020 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T10:57:29.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:57:29.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T10:57:29.600 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T10:57:29.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:57:29.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:57:29.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:57:29.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:57:29.629 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:29.624+0000 7f43df520540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:29.632 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:29.628+0000 7f43df520540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:29.634 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:29.628+0000 7f43df520540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:29.989 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:57:29.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:57:29.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:29.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:30.174 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:30.168+0000 7f43df520540 -1 Falling back to public interface 2024-09-18T10:57:30.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:31.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:31.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:31.436 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:57:31.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:57:31.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:31.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:31.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:32.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:32.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:32.841 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:57:32.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:57:32.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:32.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:32.951 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:32.948+0000 7f43df520540 -1 osd.0 0 log_to_monitors true 2024-09-18T10:57:33.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:34.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:34.328 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:57:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:57:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:34.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:35.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:35.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:35.776 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:57:35.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:57:35.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:35.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:36.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:37.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:37.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:37.208 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:57:37.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:57:37.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:37.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:37.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:38.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:38.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:38.796 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:57:38.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:57:38.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:38.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:39.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:40.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:40.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:40.223 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:57:40.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:57:40.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:40.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:40.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:41.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:41.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:41.726 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:57:41.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:57:41.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:41.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:42.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:43.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:43.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:43.137 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:57:43.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:57:43.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:43.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:43.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:44.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:44.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:44.693 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:57:44.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:57:44.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:44.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:45.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:46.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:46.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:46.404 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:57:46.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:57:46.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:46.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:47.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:48.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:48.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:48.023 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:57:48.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:57:48.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:48.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:48.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:49.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:49.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:49.667 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:57:49.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:57:49.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:49.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:50.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:51.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:51.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:51.555 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:57:51.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:57:51.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:51.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:52.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:53.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:53.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:53.610 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:57:53.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:57:53.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:53.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:54.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:57:55.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:57:55.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:57:55.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:57:55.634 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:57:55.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:57:55.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T10:57:56.174 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/120400,v1:127.0.0.1:6803/120400] [v2:127.0.0.1:6804/120400,v1:127.0.0.1:6805/120400] exists,up 512c7035-e619-49d8-9576-574dba518018 2024-09-18T10:57:56.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:57:56.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:57:56.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:57:56.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T10:57:56.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:57:56.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:57:56.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:57:56.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T10:57:56.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:57:56.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9724b2a6-02d1-45dc-8c4f-64348710f849 2024-09-18T10:57:56.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9724b2a6-02d1-45dc-8c4f-64348710f849' 2024-09-18T10:57:56.184 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 9724b2a6-02d1-45dc-8c4f-64348710f849 2024-09-18T10:57:56.184 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:57:56.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0supmIOzaCxAApCySISfdERJcEPGkS2ib9Q== 2024-09-18T10:57:56.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0supmIOzaCxAApCySISfdERJcEPGkS2ib9Q=="}' 2024-09-18T10:57:56.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9724b2a6-02d1-45dc-8c4f-64348710f849 -i td/test-erasure-eio/1/new.json 2024-09-18T10:57:56.909 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:57:56.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T10:57:56.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0supmIOzaCxAApCySISfdERJcEPGkS2ib9Q== --osd-uuid 9724b2a6-02d1-45dc-8c4f-64348710f849 2024-09-18T10:57:56.957 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:56.951+0000 7f4db501e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:56.981 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:56.975+0000 7f4db501e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:56.983 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:56.979+0000 7f4db501e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:57:56.984 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:56.979+0000 7f4db501e540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:57:56.984 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:56.979+0000 7f4db501e540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:57:56.984 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:56.979+0000 7f4db501e540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T10:57:57.073 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:57:57.067+0000 7f4db501e540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T10:58:02.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T10:58:02.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:58:02.994 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T10:58:02.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T10:58:02.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:58:03.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T10:58:03.544 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T10:58:03.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:58:03.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:58:03.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:58:03.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:58:03.572 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:03.567+0000 7fcfd2360540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:03.567+0000 7fcfd2360540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:03.576 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:03.571+0000 7fcfd2360540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:03.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T10:58:03.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:58:03.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T10:58:03.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:58:03.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:58:03.940 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:58:03.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:03.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:58:03.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:03.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:04.137 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:04.131+0000 7fcfd2360540 -1 Falling back to public interface 2024-09-18T10:58:04.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:05.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:05.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:05.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:58:05.344 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:58:05.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:05.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:05.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:06.715 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:06.711+0000 7fcfd2360540 -1 osd.1 0 log_to_monitors true 2024-09-18T10:58:06.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:06.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:06.811 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:58:06.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:58:06.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:06.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:07.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:08.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:08.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:08.398 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:58:08.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:58:08.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:08.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:08.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:09.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:09.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:09.884 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:58:09.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:58:09.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:09.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:10.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:11.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:11.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:11.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:58:11.340 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:58:11.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:11.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:11.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:12.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:12.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:12.929 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:58:12.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:58:12.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:12.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:13.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:14.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:14.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:14.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:58:14.357 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:58:14.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:14.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:14.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:15.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:15.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:15.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:58:15.817 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:58:15.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:15.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:16.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:17.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:17.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:17.261 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:58:17.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:58:17.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:17.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:18.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:19.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:19.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:19.011 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:58:19.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:58:19.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:19.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:19.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:20.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:20.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:20.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:58:20.462 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:58:20.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:20.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:21.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:22.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:22.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:22.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:58:22.495 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:58:22.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:22.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:23.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:24.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:24.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:24.234 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:58:24.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:58:24.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:24.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:24.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:25.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:25.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:25.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:58:25.969 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:58:25.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:25.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:26.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:27.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:27.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:27.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:58:27.512 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:58:27.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:27.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:28.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:29.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:29.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:29.542 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:58:29.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:58:29.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:29.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:29.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:30.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:30.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:30.958 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:58:30.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:58:30.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:30.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T10:58:31.438 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/122149,v1:127.0.0.1:6811/122149] [v2:127.0.0.1:6812/122149,v1:127.0.0.1:6813/122149] exists,up 9724b2a6-02d1-45dc-8c4f-64348710f849 2024-09-18T10:58:31.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:58:31.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:58:31.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:58:31.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:58:31.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T10:58:31.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T10:58:31.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:58:31.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:58:31.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:58:31.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:58:31.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:58:31.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:58:31.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:58:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T10:58:31.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:58:31.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b76c0176-a92b-419c-8282-caf24481a9d3 2024-09-18T10:58:31.446 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 b76c0176-a92b-419c-8282-caf24481a9d3 2024-09-18T10:58:31.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b76c0176-a92b-419c-8282-caf24481a9d3' 2024-09-18T10:58:31.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:58:31.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXsupmPmeFGxAAWpYqFMBQdNy5ner/XcvFkA== 2024-09-18T10:58:31.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXsupmPmeFGxAAWpYqFMBQdNy5ner/XcvFkA=="}' 2024-09-18T10:58:31.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b76c0176-a92b-419c-8282-caf24481a9d3 -i td/test-erasure-eio/2/new.json 2024-09-18T10:58:32.026 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:58:32.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T10:58:32.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXsupmPmeFGxAAWpYqFMBQdNy5ner/XcvFkA== --osd-uuid b76c0176-a92b-419c-8282-caf24481a9d3 2024-09-18T10:58:32.076 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.071+0000 7f8803cc2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:32.079 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.075+0000 7f8803cc2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:32.081 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.075+0000 7f8803cc2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:32.082 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.075+0000 7f8803cc2540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:58:32.082 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.075+0000 7f8803cc2540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:58:32.083 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.075+0000 7f8803cc2540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T10:58:32.196 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:32.191+0000 7f8803cc2540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T10:58:38.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T10:58:38.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:58:38.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T10:58:38.536 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T10:58:38.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:58:39.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T10:58:39.116 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T10:58:39.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:58:39.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:58:39.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:58:39.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:58:39.144 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:39.139+0000 7f57ff81f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:39.143+0000 7f57ff81f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:39.154 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:39.147+0000 7f57ff81f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:58:39.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T10:58:39.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:58:39.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T10:58:39.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:58:39.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:58:39.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:39.529 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:58:39.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:58:39.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:39.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:39.725 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:39.719+0000 7f57ff81f540 -1 Falling back to public interface 2024-09-18T10:58:39.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:40.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:40.949 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:58:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:58:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:40.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:41.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:42.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:42.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:42.361 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:58:42.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:58:42.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:42.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:42.440 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:58:42.435+0000 7f57ff81f540 -1 osd.2 0 log_to_monitors true 2024-09-18T10:58:42.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:43.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:43.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:43.882 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:58:43.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:58:43.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:43.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:44.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:45.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:45.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:45.352 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:58:45.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:58:45.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:45.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:45.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:46.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:46.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:46.896 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:58:46.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:58:46.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:46.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:47.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:48.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:48.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:48.426 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:58:48.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:58:48.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:48.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:48.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:49.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:49.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:49.859 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:58:49.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:58:49.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:49.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:50.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:51.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:51.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:51.393 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:58:51.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:58:51.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:51.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:51.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:52.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:52.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:52.832 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:58:52.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:58:52.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:52.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:53.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:54.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:54.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:54.362 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:58:54.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:58:54.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:54.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:54.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:55.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:55.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:55.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:58:55.836 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:58:55.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:55.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:56.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:57.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:57.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:57.691 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:58:57.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:58:57.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:57.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:58:58.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:58:59.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:58:59.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:58:59.858 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:58:59.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:58:59.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:58:59.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:59:00.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:01.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:01.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:01.543 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:59:01.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:59:01.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:01.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:59:02.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:03.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:03.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:03.648 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:59:03.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:59:03.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:03.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:59:04.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:05.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:05.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:05.453 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:59:05.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:59:05.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:05.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:59:06.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:07.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:07.014 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:59:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:59:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:07.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T10:59:07.755 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/123924,v1:127.0.0.1:6819/123924] [v2:127.0.0.1:6820/123924,v1:127.0.0.1:6821/123924] exists,up b76c0176-a92b-419c-8282-caf24481a9d3 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:59:07.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:59:07.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:59:07.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:59:07.761 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:59:07.761 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:59:07.761 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:59:07.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:59:07.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T10:59:07.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:59:07.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=350d6902-420b-4e03-b5b6-3ab6ce52699f 2024-09-18T10:59:07.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 350d6902-420b-4e03-b5b6-3ab6ce52699f' 2024-09-18T10:59:07.764 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 350d6902-420b-4e03-b5b6-3ab6ce52699f 2024-09-18T10:59:07.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:59:07.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7supmWBe6LhAAvF4lRd/dCsGebmh4ya6T0g== 2024-09-18T10:59:07.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7supmWBe6LhAAvF4lRd/dCsGebmh4ya6T0g=="}' 2024-09-18T10:59:07.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 350d6902-420b-4e03-b5b6-3ab6ce52699f -i td/test-erasure-eio/3/new.json 2024-09-18T10:59:08.855 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:59:08.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T10:59:08.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB7supmWBe6LhAAvF4lRd/dCsGebmh4ya6T0g== --osd-uuid 350d6902-420b-4e03-b5b6-3ab6ce52699f 2024-09-18T10:59:08.908 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:08.903+0000 7f349bdd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:08.931 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:08.927+0000 7f349bdd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:08.933 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:08.927+0000 7f349bdd1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:08.934 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:08.927+0000 7f349bdd1540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:59:08.934 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:08.927+0000 7f349bdd1540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:59:08.934 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:08.927+0000 7f349bdd1540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T10:59:09.072 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:09.067+0000 7f349bdd1540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T10:59:15.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T10:59:15.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:59:15.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T10:59:15.454 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T10:59:15.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:59:16.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T10:59:16.003 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T10:59:16.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:59:16.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:59:16.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:59:16.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:59:16.028 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:16.022+0000 7fba5d5f6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:16.065 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:16.058+0000 7fba5d5f6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:16.067 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:16.062+0000 7fba5d5f6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:16.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:59:16.448 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:59:16.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:16.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:16.646 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:16.638+0000 7fba5d5f6540 -1 Falling back to public interface 2024-09-18T10:59:16.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:17.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:17.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:17.875 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:59:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:59:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:18.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:19.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:19.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:19.268 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:59:19.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:59:19.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:19.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:19.454 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:19.446+0000 7fba5d5f6540 -1 osd.3 0 log_to_monitors true 2024-09-18T10:59:19.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:20.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:20.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:20.784 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:59:20.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:59:20.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:20.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:21.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:22.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:22.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:22.426 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:59:22.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T10:59:22.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:22.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:24.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:24.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:24.181 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T10:59:24.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T10:59:24.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:24.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:24.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:25.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:25.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:25.741 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T10:59:25.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T10:59:25.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:25.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:26.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:27.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:27.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:27.240 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T10:59:27.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T10:59:27.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:27.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:27.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:28.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:28.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:28.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T10:59:28.779 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T10:59:28.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:28.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:29.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:30.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:30.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:30.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T10:59:30.237 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T10:59:30.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:30.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:30.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:31.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:31.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:31.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T10:59:31.826 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T10:59:31.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:31.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:32.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:33.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:33.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:33.238 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T10:59:33.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T10:59:33.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:33.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:34.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:35.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:35.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:35.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T10:59:35.170 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T10:59:35.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:35.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:35.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:36.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:36.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:36.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T10:59:36.819 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T10:59:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:37.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:38.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:38.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:38.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T10:59:38.576 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T10:59:38.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:38.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:39.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:40.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:40.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:40.545 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T10:59:40.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T10:59:40.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:40.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:41.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:42.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T10:59:42.342 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T10:59:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:43.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:44.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:44.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:44.408 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T10:59:44.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T10:59:44.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:44.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T10:59:44.845 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/125702,v1:127.0.0.1:6827/125702] [v2:127.0.0.1:6828/125702,v1:127.0.0.1:6829/125702] exists,up 350d6902-420b-4e03-b5b6-3ab6ce52699f 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T10:59:44.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T10:59:44.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T10:59:44.850 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T10:59:44.850 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T10:59:44.850 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T10:59:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T10:59:44.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca8b8f14-9a77-49ee-8293-f0d029cd9423 2024-09-18T10:59:44.853 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 ca8b8f14-9a77-49ee-8293-f0d029cd9423 2024-09-18T10:59:44.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 ca8b8f14-9a77-49ee-8293-f0d029cd9423' 2024-09-18T10:59:44.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T10:59:44.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgsupmu9LRMxAAXV7FHFOtN8PNFGgu38uqBA== 2024-09-18T10:59:44.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgsupmu9LRMxAAXV7FHFOtN8PNFGgu38uqBA=="}' 2024-09-18T10:59:44.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca8b8f14-9a77-49ee-8293-f0d029cd9423 -i td/test-erasure-eio/4/new.json 2024-09-18T10:59:45.538 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T10:59:45.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T10:59:45.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgsupmu9LRMxAAXV7FHFOtN8PNFGgu38uqBA== --osd-uuid ca8b8f14-9a77-49ee-8293-f0d029cd9423 2024-09-18T10:59:45.586 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.578+0000 7fc775a1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:45.638 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.630+0000 7fc775a1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:45.640 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.634+0000 7fc775a1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:45.642 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.634+0000 7fc775a1d540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:59:45.642 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.634+0000 7fc775a1d540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:59:45.642 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.634+0000 7fc775a1d540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T10:59:45.787 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:45.782+0000 7fc775a1d540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T10:59:53.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T10:59:53.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T10:59:53.286 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T10:59:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T10:59:53.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T10:59:53.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T10:59:53.875 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T10:59:53.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T10:59:53.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T10:59:53.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T10:59:53.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T10:59:53.904 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:53.898+0000 7f9d1c294540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:53.906 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:53.898+0000 7f9d1c294540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:53.907 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:53.902+0000 7f9d1c294540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T10:59:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T10:59:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T10:59:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T10:59:54.320 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T10:59:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T10:59:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T10:59:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T10:59:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:59:54.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:54.465 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:54.458+0000 7f9d1c294540 -1 Falling back to public interface 2024-09-18T10:59:54.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:55.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:55.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:55.769 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T10:59:55.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T10:59:55.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:55.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:59:56.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:57.114 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T10:59:57.106+0000 7f9d1c294540 -1 osd.4 0 log_to_monitors true 2024-09-18T10:59:57.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:57.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:57.164 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T10:59:57.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T10:59:57.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:57.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:59:57.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T10:59:58.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T10:59:58.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T10:59:58.726 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T10:59:58.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T10:59:58.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T10:59:58.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T10:59:59.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:00.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:00.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:00.277 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:00:00.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:00:00.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:00.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:00.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:01.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:01.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:01.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:00:01.760 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:00:01.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:01.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:02.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:03.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:03.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:03.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:00:03.345 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:00:03.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:03.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:04.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:04.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:04.833 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:00:04.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:00:04.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:04.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:05.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:06.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:06.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:06.322 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:00:06.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:00:06.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:06.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:06.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:07.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:07.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:07.721 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:00:07.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:00:07.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:07.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:08.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:09.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:09.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:09.153 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:00:09.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:00:09.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:09.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:09.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:10.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:10.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:10.699 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:00:10.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:00:10.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:10.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:11.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:12.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:12.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:12.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:00:12.213 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:00:12.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:12.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:13.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:14.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:14.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:14.031 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:00:14.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:00:14.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:14.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:14.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:15.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:15.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:15.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:00:15.888 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:00:15.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:15.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:16.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:17.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:17.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:17.761 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:00:17.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:00:17.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:17.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:18.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:19.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:19.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:19.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:00:19.543 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:00:19.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:19.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:20.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:21.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:21.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:21.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:00:21.180 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:00:21.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:21.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:21.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:22.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:22.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:22.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:00:22.815 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:00:22.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:22.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:00:23.519 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/127475,v1:127.0.0.1:6835/127475] [v2:127.0.0.1:6836/127475,v1:127.0.0.1:6837/127475] exists,up ca8b8f14-9a77-49ee-8293-f0d029cd9423 2024-09-18T11:00:23.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:00:23.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:00:23.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-18T11:00:23.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-18T11:00:23.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:00:23.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:00:23.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:00:23.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:00:23.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:00:23.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:00:23.523 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:00:23.523 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:00:23.523 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:00:23.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:00:23.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:00:23.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-18T11:00:23.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:00:23.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63ee55f1-290d-42d2-a3f0-570c58ce2f2a 2024-09-18T11:00:23.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 63ee55f1-290d-42d2-a3f0-570c58ce2f2a' 2024-09-18T11:00:23.527 INFO:tasks.workunit.client.0.smithi174.stdout:add osd5 63ee55f1-290d-42d2-a3f0-570c58ce2f2a 2024-09-18T11:00:23.527 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:00:23.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDHsupmcHRrIBAAH1aZQVrS0LITnoANifnnxw== 2024-09-18T11:00:23.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDHsupmcHRrIBAAH1aZQVrS0LITnoANifnnxw=="}' 2024-09-18T11:00:23.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63ee55f1-290d-42d2-a3f0-570c58ce2f2a -i td/test-erasure-eio/5/new.json 2024-09-18T11:00:24.804 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:00:24.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-18T11:00:24.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHsupmcHRrIBAAH1aZQVrS0LITnoANifnnxw== --osd-uuid 63ee55f1-290d-42d2-a3f0-570c58ce2f2a 2024-09-18T11:00:24.857 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.850+0000 7f81c625e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:00:24.868 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.862+0000 7f81c625e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:00:24.870 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.862+0000 7f81c625e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:00:24.870 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.862+0000 7f81c625e540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:00:24.870 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.862+0000 7f81c625e540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:00:24.871 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.862+0000 7f81c625e540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:00:24.979 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:24.974+0000 7f81c625e540 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-18T11:00:32.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-18T11:00:32.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:00:32.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T11:00:32.010 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd5 key to auth repository 2024-09-18T11:00:32.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:00:32.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T11:00:32.590 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.5 2024-09-18T11:00:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:00:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:00:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:00:32.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:00:32.614 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:32.605+0000 7f597a70d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:00:32.627 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:32.621+0000 7f597a70d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:00:32.629 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:32.621+0000 7f597a70d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:33.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:00:33.020 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:00:33.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:33.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:33.189 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:33.181+0000 7f597a70d540 -1 Falling back to public interface 2024-09-18T11:00:33.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:34.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:34.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:34.463 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:00:34.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:00:34.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:34.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:34.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:35.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:35.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:35.853 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:00:35.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:00:35.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:35.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:36.002 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:00:35.993+0000 7f597a70d540 -1 osd.5 0 log_to_monitors true 2024-09-18T11:00:36.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:37.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:37.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:37.355 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:00:37.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:00:37.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:37.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:38.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:39.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:39.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:39.112 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:00:39.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:00:39.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:39.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:39.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:40.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:40.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:40.750 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:00:40.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:00:40.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:40.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:41.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:42.292 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:00:42.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:00:42.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:42.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:42.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:43.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:43.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:43.731 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:00:43.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:00:43.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:43.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:44.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:45.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:45.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:45.212 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:00:45.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:00:45.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:45.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:45.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:46.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:46.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:46.755 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:00:46.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:00:46.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:46.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:47.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:48.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:48.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:48.293 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:00:48.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:00:48.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:48.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:49.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:49.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:49.722 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:00:49.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:00:49.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:49.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:50.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:51.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:51.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:51.207 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:00:51.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:00:51.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:51.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:51.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:52.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:52.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:52.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:00:52.746 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:00:52.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:52.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:53.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:54.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:54.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:54.754 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:00:54.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:00:54.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:54.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:55.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:56.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:56.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:56.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:00:56.697 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:00:56.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:56.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:57.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:00:58.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:00:58.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:00:58.381 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:00:58.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:00:58.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:00:58.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:00:59.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:00.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:00.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:00.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:01:00.393 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:01:00.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:00.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:01:00.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:01.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:01.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:01.949 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:01:01.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:01:01.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:01.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:01:02.766 INFO:tasks.workunit.client.0.smithi174.stdout:osd.5 up in weight 1 up_from 48 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/129278,v1:127.0.0.1:6843/129278] [v2:127.0.0.1:6844/129278,v1:127.0.0.1:6845/129278] exists,up 63ee55f1-290d-42d2-a3f0-570c58ce2f2a 2024-09-18T11:01:02.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:01:02.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:01:02.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:01:02.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:01:02.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-18T11:01:02.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-18T11:01:02.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:01:02.771 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:01:02.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:01:02.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:01:02.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:01:02.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-18T11:01:02.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:01:02.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=16dba7dc-5526-4746-9af9-26814f6cfd7e 2024-09-18T11:01:02.775 INFO:tasks.workunit.client.0.smithi174.stdout:add osd6 16dba7dc-5526-4746-9af9-26814f6cfd7e 2024-09-18T11:01:02.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 16dba7dc-5526-4746-9af9-26814f6cfd7e' 2024-09-18T11:01:02.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:01:02.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDusupm/PhjLxAARb+t/itlFBCSIKDXhmDWVg== 2024-09-18T11:01:02.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDusupm/PhjLxAARb+t/itlFBCSIKDXhmDWVg=="}' 2024-09-18T11:01:02.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 16dba7dc-5526-4746-9af9-26814f6cfd7e -i td/test-erasure-eio/6/new.json 2024-09-18T11:01:03.620 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:01:03.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-18T11:01:03.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDusupm/PhjLxAARb+t/itlFBCSIKDXhmDWVg== --osd-uuid 16dba7dc-5526-4746-9af9-26814f6cfd7e 2024-09-18T11:01:03.670 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.661+0000 7fddcbfd0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:03.673 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.665+0000 7fddcbfd0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:03.675 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.669+0000 7fddcbfd0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:03.676 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.669+0000 7fddcbfd0540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:01:03.676 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.669+0000 7fddcbfd0540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:01:03.676 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.669+0000 7fddcbfd0540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:01:03.846 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:03.841+0000 7fddcbfd0540 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-18T11:01:10.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-18T11:01:10.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:01:10.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T11:01:10.402 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd6 key to auth repository 2024-09-18T11:01:10.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:01:10.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T11:01:10.983 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.6 2024-09-18T11:01:10.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:01:10.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:01:10.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:01:10.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:01:11.014 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:11.009+0000 7f1f8373d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:11.017 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:11.009+0000 7f1f8373d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:11.018 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:11.009+0000 7f1f8373d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:01:11.381 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:01:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:11.601 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:11.593+0000 7f1f8373d540 -1 Falling back to public interface 2024-09-18T11:01:11.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:01:12.763 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:01:12.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:12.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:13.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:14.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:14.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:14.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:01:14.156 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:01:14.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:14.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:14.292 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:14.285+0000 7f1f8373d540 -1 osd.6 0 log_to_monitors true 2024-09-18T11:01:14.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:15.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:15.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:15.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:01:15.690 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:01:15.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:15.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:16.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:17.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:17.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:17.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:01:17.247 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:01:17.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:17.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:17.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:01:18.961 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:01:18.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:18.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:19.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:20.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:20.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:20.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:01:20.528 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:01:20.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:20.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:20.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:21.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:21.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:21.971 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:01:21.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:01:21.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:21.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:22.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:23.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:23.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:23.470 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:01:23.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:01:23.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:23.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:24.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:25.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:25.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:25.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:01:25.013 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:01:25.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:25.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:25.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:26.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:26.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:26.508 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:01:26.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:01:26.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:26.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:26.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:27.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:27.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:27.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:01:27.949 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:01:27.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:27.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:28.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:29.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:29.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:29.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:01:29.626 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:01:29.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:29.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:30.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:31.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:31.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:31.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:01:31.189 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:01:31.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:31.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:31.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:32.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:32.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:32.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:01:32.745 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:01:32.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:32.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:33.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:34.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:34.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:34.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:01:34.697 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:01:34.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:34.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:35.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:36.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:36.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:36.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:01:36.312 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:01:36.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:36.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:37.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:38.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:38.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:38.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:01:38.117 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:01:38.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:38.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:39.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:40.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:40.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:40.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:01:40.096 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:01:40.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:40.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:01:40.528 INFO:tasks.workunit.client.0.smithi174.stdout:osd.6 up in weight 1 up_from 58 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/131079,v1:127.0.0.1:6851/131079] [v2:127.0.0.1:6852/131079,v1:127.0.0.1:6853/131079] exists,up 16dba7dc-5526-4746-9af9-26814f6cfd7e 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:01:40.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:01:40.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:01:40.533 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:01:40.533 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:01:40.533 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:01:40.533 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:01:40.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:01:40.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c7ecba4a-87da-435c-9698-1f4b26be3755 2024-09-18T11:01:40.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 c7ecba4a-87da-435c-9698-1f4b26be3755' 2024-09-18T11:01:40.536 INFO:tasks.workunit.client.0.smithi174.stdout:add osd7 c7ecba4a-87da-435c-9698-1f4b26be3755 2024-09-18T11:01:40.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:01:40.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUs+pmnmnOIBAAJqcasm/lRyNl5gHTR4CEwA== 2024-09-18T11:01:40.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUs+pmnmnOIBAAJqcasm/lRyNl5gHTR4CEwA=="}' 2024-09-18T11:01:40.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c7ecba4a-87da-435c-9698-1f4b26be3755 -i td/test-erasure-eio/7/new.json 2024-09-18T11:01:41.515 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:01:41.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-09-18T11:01:41.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUs+pmnmnOIBAAJqcasm/lRyNl5gHTR4CEwA== --osd-uuid c7ecba4a-87da-435c-9698-1f4b26be3755 2024-09-18T11:01:41.561 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.553+0000 7f8d60867540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:41.564 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.557+0000 7f8d60867540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:41.566 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.557+0000 7f8d60867540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:41.567 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.561+0000 7f8d60867540 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-18T11:01:41.567 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.561+0000 7f8d60867540 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-18T11:01:41.567 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.561+0000 7f8d60867540 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-09-18T11:01:41.682 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:41.673+0000 7f8d60867540 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-09-18T11:01:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-09-18T11:01:47.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:01:47.913 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd7 key to auth repository 2024-09-18T11:01:47.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-09-18T11:01:47.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:01:48.462 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.7 2024-09-18T11:01:48.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-09-18T11:01:48.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:01:48.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:01:48.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:01:48.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:01:48.493 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:48.485+0000 7f612e5b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:48.527 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:48.521+0000 7f612e5b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:48.528 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:48.521+0000 7f612e5b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:01:48.828 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:48.821+0000 7f612e5b9540 -1 Falling back to public interface 2024-09-18T11:01:48.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-09-18T11:01:48.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:01:48.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-09-18T11:01:48.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:01:48.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:01:48.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:48.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:01:48.859 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:01:48.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:48.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:49.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:50.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:50.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:50.239 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:01:50.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:01:50.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:50.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:50.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:51.629 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:01:51.621+0000 7f612e5b9540 -1 osd.7 0 log_to_monitors true 2024-09-18T11:01:51.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:51.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:51.632 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:01:51.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:01:51.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:51.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:52.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:53.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:53.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:53.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:01:53.236 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:01:53.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:53.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:53.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:54.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:54.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:54.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:01:54.954 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:01:54.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:54.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:55.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:56.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:56.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:56.472 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:01:56.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:01:56.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:56.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:56.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:57.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:57.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:57.941 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:01:57.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:01:57.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:57.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:58.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:01:59.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:01:59.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:01:59.421 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:01:59.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:01:59.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:01:59.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:01:59.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:00.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:00.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:00.937 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:02:00.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:02:00.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:00.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:01.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:02.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:02.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:02.429 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:02:02.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:02:02.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:02.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:02.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:03.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:03.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:03.837 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:02:03.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:02:03.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:03.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:04.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:05.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:05.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:05.264 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:02:05.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:02:05.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:05.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:05.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:06.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:06.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:06.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:02:06.928 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:02:06.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:06.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:07.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:08.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:08.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:08.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:02:08.577 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:02:08.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:08.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:09.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:10.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:10.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:10.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:02:10.338 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:02:10.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:10.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:11.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:12.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:02:12.108 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:02:12.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:12.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:12.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:13.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:13.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:13.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:02:13.750 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:02:13.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:13.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:14.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:15.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:15.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:15.867 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:02:15.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:02:15.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:15.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:16.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:17.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:17.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:17.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:02:17.323 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:02:17.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:17.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-09-18T11:02:17.858 INFO:tasks.workunit.client.0.smithi174.stdout:osd.7 up in weight 1 up_from 68 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/132884,v1:127.0.0.1:6859/132884] [v2:127.0.0.1:6860/132884,v1:127.0.0.1:6861/132884] exists,up c7ecba4a-87da-435c-9698-1f4b26be3755 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:02:17.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:02:17.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:02:17.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:02:17.864 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:02:17.864 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:02:17.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:02:17.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-09-18T11:02:17.866 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:02:17.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2681fa7-d22e-4937-8619-31241ea0a816 2024-09-18T11:02:17.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 e2681fa7-d22e-4937-8619-31241ea0a816' 2024-09-18T11:02:17.867 INFO:tasks.workunit.client.0.smithi174.stdout:add osd8 e2681fa7-d22e-4937-8619-31241ea0a816 2024-09-18T11:02:17.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:02:17.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5s+pm84zPNBAAm/8dbNver3s7dXbbaA4vQw== 2024-09-18T11:02:17.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5s+pm84zPNBAAm/8dbNver3s7dXbbaA4vQw=="}' 2024-09-18T11:02:17.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2681fa7-d22e-4937-8619-31241ea0a816 -i td/test-erasure-eio/8/new.json 2024-09-18T11:02:18.849 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:02:18.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-09-18T11:02:18.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5s+pm84zPNBAAm/8dbNver3s7dXbbaA4vQw== --osd-uuid e2681fa7-d22e-4937-8619-31241ea0a816 2024-09-18T11:02:18.899 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:18.896+0000 7f4d71d94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:02:18.902 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:18.896+0000 7f4d71d94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:02:18.904 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:18.900+0000 7f4d71d94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:02:18.905 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:18.900+0000 7f4d71d94540 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-18T11:02:18.905 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:18.900+0000 7f4d71d94540 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-18T11:02:18.905 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:18.900+0000 7f4d71d94540 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-09-18T11:02:19.056 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:19.052+0000 7f4d71d94540 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-09-18T11:02:25.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-09-18T11:02:25.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:02:25.430 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd8 key to auth repository 2024-09-18T11:02:25.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-09-18T11:02:25.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:02:25.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-09-18T11:02:25.963 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.8 2024-09-18T11:02:25.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:02:25.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:02:25.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:02:25.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:02:25.991 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:25.988+0000 7fb47aa1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:02:25.993 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:25.988+0000 7fb47aa1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:02:25.995 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:25.992+0000 7fb47aa1d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:02:26.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-09-18T11:02:26.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:02:26.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-09-18T11:02:26.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:02:26.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:02:26.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:26.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:02:26.351 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:02:26.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:26.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:26.549 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:26.544+0000 7fb47aa1d540 -1 Falling back to public interface 2024-09-18T11:02:26.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:27.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:27.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:27.781 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:02:27.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:02:27.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:27.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:28.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:29.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:29.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:29.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:02:29.172 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:02:29.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:29.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:29.368 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:02:29.364+0000 7fb47aa1d540 -1 osd.8 0 log_to_monitors true 2024-09-18T11:02:29.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:30.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:30.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:30.582 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:02:30.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:02:30.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:30.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:31.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:32.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:32.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:32.066 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:02:32.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:02:32.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:32.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:32.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:33.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:33.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:33.709 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:02:33.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:02:33.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:33.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:34.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:35.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:35.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:35.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:02:35.439 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:02:35.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:35.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:35.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:36.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:36.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:36.853 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:02:36.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:02:36.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:36.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:37.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:38.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:38.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:38.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:02:38.436 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:02:38.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:38.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:38.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:39.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:39.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:39.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:02:39.954 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:02:39.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:39.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:40.422 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:41.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:41.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:41.424 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:02:41.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:02:41.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:41.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:41.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:42.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:42.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:42.845 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:02:42.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:02:42.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:42.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:43.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:44.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:02:44.344 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:02:44.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:44.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:45.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:46.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:46.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:46.155 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:02:46.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:02:46.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:46.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:46.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:47.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:47.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:47.995 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:02:47.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:02:47.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:47.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:48.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:49.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:49.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:49.608 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:02:49.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:02:49.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:49.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:50.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:51.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:51.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:51.696 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:02:51.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:02:51.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:51.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:52.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:53.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:53.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:53.218 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:02:53.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:02:53.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:53.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:53.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:02:54.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:02:54.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:02:54.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:02:54.706 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:02:54.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:02:54.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-09-18T11:02:55.227 INFO:tasks.workunit.client.0.smithi174.stdout:osd.8 up in weight 1 up_from 77 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/134683,v1:127.0.0.1:6867/134683] [v2:127.0.0.1:6868/134683,v1:127.0.0.1:6869/134683] exists,up e2681fa7-d22e-4937-8619-31241ea0a816 2024-09-18T11:02:55.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:02:55.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:02:55.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:02:55.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:02:55.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:02:55.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:02:55.229 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:02:55.230 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:02:55.230 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:02:55.230 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:02:55.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:02:55.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:02:55.319 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:02:55.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:02:55.340 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T10:57:30.180+0000 7f43df520540 0 load: jerasure load: lrc 2024-09-18T11:02:55.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:02:55.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-09-18T11:02:56.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:02:56.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:02:58.318 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:02:58.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:02:59.332 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:02:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:02:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:02:59.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:02:59.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:02:59.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:02:59.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:02:59.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:02:59.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:03:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:03:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:03:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:03:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:03:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:8' 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:00.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:03:00.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803846 2024-09-18T11:03:00.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803846 2024-09-18T11:03:00.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846' 2024-09-18T11:03:00.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:00.347 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:03:00.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574910 2024-09-18T11:03:00.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574910 2024-09-18T11:03:00.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910' 2024-09-18T11:03:00.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:00.488 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:03:00.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378680 2024-09-18T11:03:00.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378680 2024-09-18T11:03:00.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680' 2024-09-18T11:03:00.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:00.660 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:03:00.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084340 2024-09-18T11:03:00.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084340 2024-09-18T11:03:00.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680 3-120259084340' 2024-09-18T11:03:00.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:00.798 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:03:00.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724589 2024-09-18T11:03:00.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724589 2024-09-18T11:03:00.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680 3-120259084340 4-167503724589' 2024-09-18T11:03:00.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:00.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:03:01.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=206158430246 2024-09-18T11:03:01.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 206158430246 2024-09-18T11:03:01.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680 3-120259084340 4-167503724589 5-206158430246' 2024-09-18T11:03:01.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:01.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:03:01.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=249108103199 2024-09-18T11:03:01.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 249108103199 2024-09-18T11:03:01.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680 3-120259084340 4-167503724589 5-206158430246 6-249108103199' 2024-09-18T11:03:01.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:01.248 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T11:03:01.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=292057776151 2024-09-18T11:03:01.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 292057776151 2024-09-18T11:03:01.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680 3-120259084340 4-167503724589 5-206158430246 6-249108103199 7-292057776151' 2024-09-18T11:03:01.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:01.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T11:03:01.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481806 2024-09-18T11:03:01.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481806 2024-09-18T11:03:01.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803846 1-55834574910 2-81604378680 3-120259084340 4-167503724589 5-206158430246 6-249108103199 7-292057776151 8-330712481806' 2024-09-18T11:03:01.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:01.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803846 2024-09-18T11:03:01.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:01.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:03:01.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803846 2024-09-18T11:03:01.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:01.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803846 2024-09-18T11:03:01.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803846' 2024-09-18T11:03:01.591 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803846 2024-09-18T11:03:01.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:03:02.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803846 -lt 25769803846 2024-09-18T11:03:02.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:02.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574910 2024-09-18T11:03:02.010 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:02.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:03:02.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574910 2024-09-18T11:03:02.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:02.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574910 2024-09-18T11:03:02.016 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574910 2024-09-18T11:03:02.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574910' 2024-09-18T11:03:02.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:03:02.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574910 -lt 55834574910 2024-09-18T11:03:02.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:02.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378680 2024-09-18T11:03:02.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:02.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:03:02.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378680 2024-09-18T11:03:02.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:02.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378680 2024-09-18T11:03:02.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378680' 2024-09-18T11:03:02.401 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378680 2024-09-18T11:03:02.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:03:02.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378680 -lt 81604378680 2024-09-18T11:03:02.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:02.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084340 2024-09-18T11:03:02.862 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:02.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:03:02.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:02.865 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084340 2024-09-18T11:03:02.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084340 2024-09-18T11:03:02.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084340' 2024-09-18T11:03:02.866 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084340 2024-09-18T11:03:02.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:03:03.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084340 -lt 120259084340 2024-09-18T11:03:03.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:03.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724589 2024-09-18T11:03:03.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:03.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:03:03.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724589 2024-09-18T11:03:03.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:03.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724589 2024-09-18T11:03:03.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724589' 2024-09-18T11:03:03.257 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724589 2024-09-18T11:03:03.257 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:03:03.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724589 -lt 167503724589 2024-09-18T11:03:03.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:03.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-206158430246 2024-09-18T11:03:03.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:03.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:03:03.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-206158430246 2024-09-18T11:03:03.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:03.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=206158430246 2024-09-18T11:03:03.794 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 206158430246 2024-09-18T11:03:03.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 206158430246' 2024-09-18T11:03:03.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:03:04.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 206158430247 -lt 206158430246 2024-09-18T11:03:04.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:04.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-249108103199 2024-09-18T11:03:04.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:04.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:03:04.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-249108103199 2024-09-18T11:03:04.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:04.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=249108103199 2024-09-18T11:03:04.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 249108103199' 2024-09-18T11:03:04.247 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 249108103199 2024-09-18T11:03:04.248 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:03:04.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 249108103199 -lt 249108103199 2024-09-18T11:03:04.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:04.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 7-292057776151 2024-09-18T11:03:04.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:04.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=7 2024-09-18T11:03:04.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 7-292057776151 2024-09-18T11:03:04.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:04.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=292057776151 2024-09-18T11:03:04.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.7 seq 292057776151' 2024-09-18T11:03:04.709 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.7 seq 292057776151 2024-09-18T11:03:04.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-18T11:03:05.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 292057776152 -lt 292057776151 2024-09-18T11:03:05.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:05.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-330712481806 2024-09-18T11:03:05.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:05.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T11:03:05.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-330712481806 2024-09-18T11:03:05.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:05.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481806 2024-09-18T11:03:05.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 330712481806' 2024-09-18T11:03:05.122 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 330712481806 2024-09-18T11:03:05.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T11:03:05.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481806 -lt 330712481806 2024-09-18T11:03:05.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:03:05.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:03:05.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:03:06.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:03:06.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:03:06.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:03:06.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:03:06.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:03:06.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:03:06.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:03:06.055 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:03:06.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:03:06.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:03:06.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:03:06.427 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:03:06.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:03:06.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:03:06.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:03:06.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:03:06.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-18T11:03:07.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:03:07.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:03:07.208 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:03:07.208 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:03:07.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:03:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:03:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:03:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:03:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T11:03:07.675 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:03:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:03:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:7' 2024-09-18T11:03:07.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-18T11:03:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:03:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T11:03:07.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:03:07.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T11:03:07.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:03:07.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T11:03:07.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:03:07.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T11:03:07.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:03:07.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:03:07.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:03:07.679 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T11:03:07.679 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:03:07.679 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:03:07.680 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:07.680 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:07.680 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:07.681 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:03:07.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:03:07.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:03:07.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:03:07.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:03:07.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T11:03:07.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:03:07.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:03:07.777 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:07.778 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:07.778 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:07.778 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:03:07.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:03:07.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:03:07.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=3 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:03:07.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:03:07.872 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:03:07.872 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:03:08.241 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:03:08.242 INFO:tasks.workunit.client.0.smithi174.stderr:7' 2024-09-18T11:03:08.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-18T11:03:08.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:03:08.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-18T11:03:08.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:03:08.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/0/type 2024-09-18T11:03:08.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:03:08.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-18T11:03:08.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:03:08.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-18T11:03:08.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:03:08.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:03:08.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:03:08.246 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-18T11:03:08.246 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:03:08.246 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:03:08.247 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:08.247 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:08.247 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:08.247 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:03:08.247 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:03:08.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:03:08.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:03:08.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:03:08.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-18T11:03:08.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:03:08.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:03:08.340 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:08.340 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:08.340 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:08.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:03:08.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:03:08.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-09-18T11:03:08.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=4 2024-09-18T11:03:08.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:03:08.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:03:08.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-18T11:03:08.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:03:08.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:03:08.434 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:03:08.434 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:03:08.845 INFO:tasks.workunit.client.0.smithi174.stderr:7' 2024-09-18T11:03:08.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-18T11:03:08.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:03:08.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=6 2024-09-18T11:03:08.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:03:08.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/6/type 2024-09-18T11:03:08.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:03:08.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-09-18T11:03:08.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:03:08.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-09-18T11:03:08.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:03:08.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:03:08.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-09-18T11:03:08.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-09-18T11:03:08.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-09-18T11:03:08.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:03:08.850 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:08.850 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:08.850 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:08.850 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.6.asok 2024-09-18T11:03:08.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:03:08.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:03:08.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:03:08.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:03:08.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.6 2024-09-18T11:03:08.939 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-09-18T11:03:08.940 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-09-18T11:03:08.941 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:08.941 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:08.941 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:08.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.6.asok 2024-09-18T11:03:08.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:03:08.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-09-18T11:03:09.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:03:09.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-09-18T11:03:09.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:03:09.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:03:09.036 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:03:09.036 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:03:09.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:03:09.442 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:03:09.442 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:03:09.442 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:7' 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-09-18T11:03:09.443 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:03:09.444 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:03:09.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:03:09.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:03:09.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:03:09.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:03:09.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-09-18T11:03:10.463 INFO:tasks.workunit.client.0.smithi174.stderr:osd.7 is already down. 2024-09-18T11:03:10.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-09-18T11:03:11.571 INFO:tasks.workunit.client.0.smithi174.stderr:osd.7 is already out. 2024-09-18T11:03:11.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-09-18T11:03:11.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:03:11.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:03:11.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:03:11.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:03:11.584 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:03:11.585 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:03:11.585 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:03:11.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:03:11.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:03:11.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:03:11.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:03:11.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:03:11.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:03:11.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:03:11.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:03:12.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:6 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:7 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:8' 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:12.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:03:12.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803851 2024-09-18T11:03:12.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803851 2024-09-18T11:03:12.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851' 2024-09-18T11:03:12.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:12.557 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:03:12.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574915 2024-09-18T11:03:12.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574915 2024-09-18T11:03:12.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915' 2024-09-18T11:03:12.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:12.715 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:03:13.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378685 2024-09-18T11:03:13.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378685 2024-09-18T11:03:13.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915 2-81604378685' 2024-09-18T11:03:13.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:13.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:03:13.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084344 2024-09-18T11:03:13.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084344 2024-09-18T11:03:13.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915 2-81604378685 3-120259084344' 2024-09-18T11:03:13.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:13.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:03:13.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724593 2024-09-18T11:03:13.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724593 2024-09-18T11:03:13.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915 2-81604378685 3-120259084344 4-167503724593' 2024-09-18T11:03:13.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:13.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:03:13.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=206158430251 2024-09-18T11:03:13.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 206158430251 2024-09-18T11:03:13.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915 2-81604378685 3-120259084344 4-167503724593 5-206158430251' 2024-09-18T11:03:13.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:13.575 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:03:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=249108103203 2024-09-18T11:03:13.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 249108103203 2024-09-18T11:03:13.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915 2-81604378685 3-120259084344 4-167503724593 5-206158430251 6-249108103203' 2024-09-18T11:03:13.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:13.727 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-18T11:03:13.826 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-18T11:03:13.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T11:03:13.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T11:03:13.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T11:03:13.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:03:13.829 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-18T11:03:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481810 2024-09-18T11:03:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481810 2024-09-18T11:03:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574915 2-81604378685 3-120259084344 4-167503724593 5-206158430251 6-249108103203 8-330712481810' 2024-09-18T11:03:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:13.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803851 2024-09-18T11:03:13.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:13.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:03:13.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803851 2024-09-18T11:03:13.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:13.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803851 2024-09-18T11:03:13.979 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803851 2024-09-18T11:03:13.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-09-18T11:03:13.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:03:14.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803849 -lt 25769803851 2024-09-18T11:03:14.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:03:15.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:03:15.383 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:03:15.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803851 -lt 25769803851 2024-09-18T11:03:15.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:15.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574915 2024-09-18T11:03:15.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:15.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:03:15.798 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574915 2024-09-18T11:03:15.799 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:15.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574915 2024-09-18T11:03:15.801 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574915 2024-09-18T11:03:15.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574915' 2024-09-18T11:03:15.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:03:16.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574915 -lt 55834574915 2024-09-18T11:03:16.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:16.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378685 2024-09-18T11:03:16.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:16.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:03:16.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378685 2024-09-18T11:03:16.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:16.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378685 2024-09-18T11:03:16.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378685' 2024-09-18T11:03:16.314 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378685 2024-09-18T11:03:16.315 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:03:16.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378685 -lt 81604378685 2024-09-18T11:03:16.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:16.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084344 2024-09-18T11:03:16.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:16.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:03:16.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084344 2024-09-18T11:03:16.797 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:16.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084344 2024-09-18T11:03:16.800 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084344 2024-09-18T11:03:16.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084344' 2024-09-18T11:03:16.800 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:03:17.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084345 -lt 120259084344 2024-09-18T11:03:17.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:17.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:17.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724593 2024-09-18T11:03:17.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:03:17.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724593 2024-09-18T11:03:17.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:17.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724593 2024-09-18T11:03:17.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724593' 2024-09-18T11:03:17.220 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724593 2024-09-18T11:03:17.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:03:17.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724594 -lt 167503724593 2024-09-18T11:03:17.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:17.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-206158430251 2024-09-18T11:03:17.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:17.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:03:17.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-206158430251 2024-09-18T11:03:17.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:17.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=206158430251 2024-09-18T11:03:17.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 206158430251' 2024-09-18T11:03:17.735 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 206158430251 2024-09-18T11:03:17.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:03:18.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 206158430251 -lt 206158430251 2024-09-18T11:03:18.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:18.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-249108103203 2024-09-18T11:03:18.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:18.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:03:18.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-249108103203 2024-09-18T11:03:18.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:18.151 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 249108103203 2024-09-18T11:03:18.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=249108103203 2024-09-18T11:03:18.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 249108103203' 2024-09-18T11:03:18.151 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:03:18.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 249108103204 -lt 249108103203 2024-09-18T11:03:18.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:03:18.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 8-330712481810 2024-09-18T11:03:18.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:03:18.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=8 2024-09-18T11:03:18.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:03:18.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 8-330712481810 2024-09-18T11:03:18.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481810 2024-09-18T11:03:18.547 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.8 seq 330712481810 2024-09-18T11:03:18.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.8 seq 330712481810' 2024-09-18T11:03:18.548 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-18T11:03:18.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481811 -lt 330712481810 2024-09-18T11:03:18.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:03:18.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:03:18.936 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:03:19.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:03:19.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:03:19.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:03:19.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:03:19.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:03:19.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:03:19.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:03:19.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:03:19.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:03:19.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:03:19.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:03:19.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:03:20.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-18T11:03:20.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:03:20.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:03:20.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:03:20.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:03:20.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:03:21.064 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:03:21.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:03:22.054 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:03:22.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:03:22.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:03:22.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:03:22.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:03:22.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:03:22.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:03:22.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:03:22.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:03:22.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:03:22.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:03:22.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:03:22.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:03:22.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:03:22.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:03:22.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:03:22.193 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:03:22.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:03:22.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:03:22.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:03:22.196 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:03:22.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:03:22.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:03:22.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:03:22.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:03:22.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:22.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:22.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:03:22.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:03:22.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:03:22.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:03:22.250 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:03:22.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:03:22.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:03:22.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:03:22.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:03:22.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:03:22.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:03:22.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:03:22.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:03:22.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:03:22.257 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:03:22.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:03:22.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:03:22.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:03:22.260 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:03:22.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:03:22.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:03:22.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:03:22.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:03:22.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:22.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:22.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:03:22.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:03:22.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:03:22.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:03:22.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:03:22.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:22.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:22.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:03:22.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:03:22.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:03:22.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:03:22.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:03:22.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:03:22.678 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:03:22.678 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:22.678 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:22.679 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:03:22.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:03:22.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:03:22.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:03:22.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:03:22.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:03:22.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:03:22.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:03:22.969 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:03:22.969 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:03:22.969 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:03:22.969 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:22.970 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:22.970 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:22.970 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:03:22.970 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:03:22.970 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:03:23.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:03:23.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:03:23.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:03:23.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:03:23.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:03:23.066 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:03:23.066 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:03:23.066 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:03:23.067 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:23.067 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:23.067 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:23.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:03:23.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:03:23.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:03:23.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:03:24.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:03:24.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:03:24.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:03:24.070 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:03:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:24.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:24.071 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:03:24.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:03:24.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:03:24.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:03:24.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:03:24.462 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:03:24.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:03:25.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:03:25.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:03:25.480 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:03:25.481 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:03:25.481 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:25.481 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:25.481 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:03:25.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:03:25.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:03:25.559 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:03:25.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:03:25.570 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:03:22.720+0000 7ff292926a00 0 load: jerasure load: lrc 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-09-18T11:03:25.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-18T11:03:25.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:03:25.572 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-18T11:03:25.573 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:03:25.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:03:25.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:03:25.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:03:25.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:03:25.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:03:25.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:03:25.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:03:25.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:03:25.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:03:25.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5bc3e1c9-b3d6-47d2-b4d3-1b7853af2f40 2024-09-18T11:03:25.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5bc3e1c9-b3d6-47d2-b4d3-1b7853af2f40' 2024-09-18T11:03:25.583 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 5bc3e1c9-b3d6-47d2-b4d3-1b7853af2f40 2024-09-18T11:03:25.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:03:25.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9s+pmPMQUJBAAo/zF17UytIyfQ912uQrV8A== 2024-09-18T11:03:25.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9s+pmPMQUJBAAo/zF17UytIyfQ912uQrV8A=="}' 2024-09-18T11:03:25.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5bc3e1c9-b3d6-47d2-b4d3-1b7853af2f40 -i td/test-erasure-eio/0/new.json 2024-09-18T11:03:25.962 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:03:25.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:03:25.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9s+pmPMQUJBAAo/zF17UytIyfQ912uQrV8A== --osd-uuid 5bc3e1c9-b3d6-47d2-b4d3-1b7853af2f40 2024-09-18T11:03:26.013 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.008+0000 7f9278040540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:03:26.017 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.012+0000 7f9278040540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:03:26.018 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.016+0000 7f9278040540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:03:26.019 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.016+0000 7f9278040540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:03:26.019 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.016+0000 7f9278040540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:03:26.019 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.016+0000 7f9278040540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:03:26.119 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:26.116+0000 7f9278040540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:03:31.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:03:31.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:03:31.613 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:03:31.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:03:31.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:03:32.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:03:32.167 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:03:32.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:03:32.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:03:32.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:03:32.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:03:32.200 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:32.195+0000 7fd98d59d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:03:32.232 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:32.227+0000 7fd98d59d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:03:32.233 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:32.227+0000 7fd98d59d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:03:32.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:03:32.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:03:32.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:03:32.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:03:32.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:03:32.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:32.630 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:03:32.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:03:32.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:32.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:32.797 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:32.791+0000 7fd98d59d540 -1 Falling back to public interface 2024-09-18T11:03:33.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:34.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:34.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:34.066 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:03:34.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:03:34.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:34.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:34.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:35.499 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:03:35.495+0000 7fd98d59d540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:03:35.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:35.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:35.528 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:03:35.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:03:35.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:35.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:35.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:36.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:36.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:36.968 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:03:36.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:03:36.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:36.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:37.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:38.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:38.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:38.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:03:38.503 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:03:38.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:38.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:38.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:39.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:39.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:39.925 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:03:39.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:03:39.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:39.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:40.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:41.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:41.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:41.358 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:03:41.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:03:41.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:41.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:41.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:42.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:42.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:42.793 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:03:42.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:03:42.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:42.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:43.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:44.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:44.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:44.294 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:03:44.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:03:44.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:44.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:44.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:45.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:45.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:45.746 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:03:45.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:03:45.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:45.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:46.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:47.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:47.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:47.250 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:03:47.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:03:47.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:47.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:47.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:48.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:48.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:48.881 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:03:48.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:03:48.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:48.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:49.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:50.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:50.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:50.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:03:50.813 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:03:50.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:50.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:51.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:52.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:52.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:52.718 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:03:52.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:03:52.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:52.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:53.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:54.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:54.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:54.228 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:03:54.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:03:54.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:54.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:54.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:55.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:55.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:55.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:03:55.701 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:03:55.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:55.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:56.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:57.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:57.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:57.251 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:03:57.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:03:57.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:57.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:57.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:03:58.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:03:58.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:03:58.917 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:03:58.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:03:58.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:03:58.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:03:59.426 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/138582,v1:127.0.0.1:6803/138582] [v2:127.0.0.1:6804/138582,v1:127.0.0.1:6805/138582] exists,up 5bc3e1c9-b3d6-47d2-b4d3-1b7853af2f40 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:03:59.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:03:59.429 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:03:59.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:03:59.432 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:03:59.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1a4a0f0-7cfa-4acf-b90c-6edd2cee60ea 2024-09-18T11:03:59.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d1a4a0f0-7cfa-4acf-b90c-6edd2cee60ea' 2024-09-18T11:03:59.433 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 d1a4a0f0-7cfa-4acf-b90c-6edd2cee60ea 2024-09-18T11:03:59.434 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:03:59.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfs+pm8bv6GhAAfHUcdO/hu0K1c2hZg416FQ== 2024-09-18T11:03:59.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfs+pm8bv6GhAAfHUcdO/hu0K1c2hZg416FQ=="}' 2024-09-18T11:03:59.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1a4a0f0-7cfa-4acf-b90c-6edd2cee60ea -i td/test-erasure-eio/1/new.json 2024-09-18T11:03:59.989 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:04:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:04:00.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfs+pm8bv6GhAAfHUcdO/hu0K1c2hZg416FQ== --osd-uuid d1a4a0f0-7cfa-4acf-b90c-6edd2cee60ea 2024-09-18T11:04:00.037 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.031+0000 7f48aaeb3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:00.041 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.035+0000 7f48aaeb3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:00.042 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.039+0000 7f48aaeb3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:00.043 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.039+0000 7f48aaeb3540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:04:00.043 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.039+0000 7f48aaeb3540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:04:00.043 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.039+0000 7f48aaeb3540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:04:00.119 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:00.115+0000 7f48aaeb3540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:04:06.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:04:06.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:04:06.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:04:06.133 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:04:06.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:04:06.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:04:06.693 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:04:06.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:04:06.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:04:06.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:04:06.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:04:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:06.723+0000 7f272a2ed540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:06.743+0000 7f272a2ed540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:06.751 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:06.747+0000 7f272a2ed540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:07.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:04:07.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:04:07.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:04:07.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:04:07.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:04:07.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:07.140 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:04:07.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:04:07.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:07.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:07.308 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:07.303+0000 7f272a2ed540 -1 Falling back to public interface 2024-09-18T11:04:07.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:08.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:08.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:08.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:04:08.634 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:04:08.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:08.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:09.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:09.909 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:09.903+0000 7f272a2ed540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:04:10.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:10.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:10.027 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:04:10.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:04:10.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:10.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:10.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:11.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:11.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:11.629 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:04:11.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:04:11.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:11.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:12.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:13.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:13.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:13.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:04:13.082 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:04:13.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:13.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:13.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:14.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:14.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:14.558 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:04:14.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:04:14.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:14.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:15.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:16.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:16.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:16.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:04:16.006 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:04:16.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:16.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:16.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:17.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:17.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:17.557 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:04:17.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:04:17.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:17.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:18.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:19.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:19.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:19.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:04:19.099 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:04:19.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:19.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:19.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:20.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:20.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:20.520 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:04:20.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:04:20.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:20.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:21.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:22.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:22.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:22.066 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:04:22.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:04:22.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:22.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:22.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:23.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:23.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:23.616 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:04:23.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:04:23.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:23.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:24.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:25.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:25.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:25.346 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:04:25.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:04:25.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:25.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:26.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:27.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:27.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:27.270 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:04:27.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:04:27.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:27.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:27.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:28.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:28.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:28.895 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:04:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:04:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:28.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:29.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:30.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:30.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:30.662 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:04:30.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:04:30.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:30.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:31.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:32.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:32.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:32.887 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:04:32.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:04:32.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:32.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:33.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:34.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:34.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:34.513 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:04:34.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:04:34.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:34.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:04:35.029 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/140355,v1:127.0.0.1:6811/140355] [v2:127.0.0.1:6812/140355,v1:127.0.0.1:6813/140355] exists,up d1a4a0f0-7cfa-4acf-b90c-6edd2cee60ea 2024-09-18T11:04:35.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:04:35.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:04:35.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:04:35.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:04:35.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:04:35.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:04:35.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:04:35.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb2a75e6-f34c-40ae-8bfe-043a1f0d59b2 2024-09-18T11:04:35.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 eb2a75e6-f34c-40ae-8bfe-043a1f0d59b2' 2024-09-18T11:04:35.037 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 eb2a75e6-f34c-40ae-8bfe-043a1f0d59b2 2024-09-18T11:04:35.038 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:04:35.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDs+pmTnCAAxAA4jqPvuYl6pqw3AITASxKlg== 2024-09-18T11:04:35.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDs+pmTnCAAxAA4jqPvuYl6pqw3AITASxKlg=="}' 2024-09-18T11:04:35.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb2a75e6-f34c-40ae-8bfe-043a1f0d59b2 -i td/test-erasure-eio/2/new.json 2024-09-18T11:04:35.699 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:04:35.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:04:35.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDs+pmTnCAAxAA4jqPvuYl6pqw3AITASxKlg== --osd-uuid eb2a75e6-f34c-40ae-8bfe-043a1f0d59b2 2024-09-18T11:04:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.747+0000 7f54a223a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:35.781 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.775+0000 7f54a223a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:35.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.779+0000 7f54a223a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:35.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.779+0000 7f54a223a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:04:35.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.779+0000 7f54a223a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:04:35.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.779+0000 7f54a223a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:04:35.897 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:35.891+0000 7f54a223a540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:04:41.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:04:41.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:04:41.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:04:41.867 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:04:41.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:04:42.404 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:04:42.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:04:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:04:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:04:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:04:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:04:42.437 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:42.431+0000 7fc52e5b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:42.455 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:42.451+0000 7fc52e5b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:42.456 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:42.451+0000 7fc52e5b5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:04:42.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:04:42.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:04:42.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:04:42.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:04:42.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:04:42.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:42.860 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:04:42.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:04:42.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:42.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:43.005 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:42.999+0000 7fc52e5b5540 -1 Falling back to public interface 2024-09-18T11:04:43.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:44.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:44.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:44.311 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:04:44.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:04:44.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:44.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:44.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:45.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:45.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:45.727 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:04:45.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:04:45.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:45.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:45.797 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:04:45.790+0000 7fc52e5b5540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:04:46.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:47.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:47.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:47.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:04:47.122 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:04:47.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:47.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:47.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:48.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:48.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:48.721 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:04:48.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:04:48.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:48.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:49.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:50.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:50.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:50.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:04:50.151 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:04:50.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:50.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:50.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:51.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:51.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:51.665 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:04:51.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:04:51.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:51.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:52.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:53.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:53.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:53.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:04:53.095 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:04:53.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:53.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:53.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:54.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:54.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:54.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:04:54.511 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:04:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:54.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:55.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:56.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:56.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:56.070 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:04:56.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:04:56.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:56.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:56.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:57.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:57.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:57.606 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:04:57.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:04:57.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:57.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:58.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:04:59.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:04:59.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:04:59.048 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:04:59.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:04:59.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:04:59.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:04:59.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:00.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:00.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:00.902 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:05:00.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:05:00.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:00.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:05:01.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:02.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:02.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:02.957 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:05:02.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:05:02.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:02.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:05:03.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:04.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:04.623 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:05:04.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:05:04.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:04.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:05:05.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:06.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:06.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:06.539 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:05:06.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:05:06.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:06.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:05:07.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:08.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:08.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:08.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:05:08.534 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:05:08.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:08.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:05:09.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:10.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:10.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:10.050 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:05:10.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:05:10.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:10.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:05:10.718 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/142130,v1:127.0.0.1:6819/142130] [v2:127.0.0.1:6820/142130,v1:127.0.0.1:6821/142130] exists,up eb2a75e6-f34c-40ae-8bfe-043a1f0d59b2 2024-09-18T11:05:10.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:05:10.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:05:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:05:10.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:05:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:05:10.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:05:10.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8caca0fd-24fd-4b41-a830-88781b3b3f45 2024-09-18T11:05:10.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 8caca0fd-24fd-4b41-a830-88781b3b3f45' 2024-09-18T11:05:10.726 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 8caca0fd-24fd-4b41-a830-88781b3b3f45 2024-09-18T11:05:10.726 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:05:10.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDms+pmN1xfLBAAo9G0QxU6Tvu65m4ZYMlSTA== 2024-09-18T11:05:10.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDms+pmN1xfLBAAo9G0QxU6Tvu65m4ZYMlSTA=="}' 2024-09-18T11:05:10.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8caca0fd-24fd-4b41-a830-88781b3b3f45 -i td/test-erasure-eio/3/new.json 2024-09-18T11:05:11.295 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:05:11.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:05:11.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDms+pmN1xfLBAAo9G0QxU6Tvu65m4ZYMlSTA== --osd-uuid 8caca0fd-24fd-4b41-a830-88781b3b3f45 2024-09-18T11:05:11.350 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.342+0000 7f4e20957540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:11.353 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.346+0000 7f4e20957540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:11.355 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.350+0000 7f4e20957540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:11.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.350+0000 7f4e20957540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:05:11.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.350+0000 7f4e20957540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:05:11.356 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.350+0000 7f4e20957540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:05:11.459 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:11.454+0000 7f4e20957540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:05:17.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:05:17.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:05:17.477 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:05:17.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:05:17.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:05:18.028 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:05:18.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:05:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:05:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:05:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:05:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:05:18.061 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:18.054+0000 7f937fe5e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:18.064 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:18.058+0000 7f937fe5e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:18.065 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:18.058+0000 7f937fe5e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:18.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:05:18.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:05:18.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:05:18.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:05:18.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:05:18.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:18.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:05:18.479 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:05:18.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:18.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:18.665 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:18.658+0000 7f937fe5e540 -1 Falling back to public interface 2024-09-18T11:05:18.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:19.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:19.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:19.913 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:05:19.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:05:19.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:19.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:20.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:21.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:21.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:21.327 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:05:21.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:05:21.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:21.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:21.409 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:21.402+0000 7f937fe5e540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:05:21.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:22.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:22.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:22.738 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:05:22.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:05:22.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:22.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:23.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:24.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:24.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:24.454 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:05:24.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:05:24.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:24.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:24.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:25.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:25.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:25.920 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:05:25.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:05:25.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:25.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:26.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:27.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:27.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:27.414 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:05:27.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:05:27.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:27.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:27.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:28.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:28.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:28.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:05:28.844 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:05:28.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:28.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:29.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:30.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:30.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:30.339 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:05:30.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:05:30.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:30.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:30.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:31.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:31.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:31.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:05:31.839 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:05:31.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:31.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:32.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:33.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:33.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:33.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:05:33.343 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:05:33.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:33.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:33.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:34.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:34.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:34.785 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:05:34.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:05:34.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:34.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:35.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:36.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:36.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:36.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:05:36.352 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:05:36.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:36.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:37.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:38.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:38.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:38.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:05:38.442 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:05:38.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:38.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:38.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:39.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:39.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:39.917 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:05:39.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:05:39.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:39.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:40.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:41.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:41.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:41.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:05:41.328 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:05:41.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:41.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:42.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:43.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:43.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:43.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:05:43.346 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:05:43.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:43.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:44.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:45.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:45.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:45.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:05:45.358 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:05:45.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:45.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:46.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:47.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:47.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:47.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:05:47.067 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:05:47.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:47.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:05:47.501 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/143908,v1:127.0.0.1:6827/143908] [v2:127.0.0.1:6828/143908,v1:127.0.0.1:6829/143908] exists,up 8caca0fd-24fd-4b41-a830-88781b3b3f45 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:05:47.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:05:47.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:05:47.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:05:47.505 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:05:47.505 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:05:47.505 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:05:47.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:05:47.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:05:47.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:05:47.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:05:47.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T11:05:47.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:05:47.508 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f7bd8d5-fbfd-4eeb-8fe9-6c34eed71899 2024-09-18T11:05:47.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0f7bd8d5-fbfd-4eeb-8fe9-6c34eed71899' 2024-09-18T11:05:47.509 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 0f7bd8d5-fbfd-4eeb-8fe9-6c34eed71899 2024-09-18T11:05:47.509 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:05:47.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALtOpm3gS6HxAAIxr/gC3q7ckV641BdFWXdQ== 2024-09-18T11:05:47.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALtOpm3gS6HxAAIxr/gC3q7ckV641BdFWXdQ=="}' 2024-09-18T11:05:47.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f7bd8d5-fbfd-4eeb-8fe9-6c34eed71899 -i td/test-erasure-eio/4/new.json 2024-09-18T11:05:48.219 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:05:48.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T11:05:48.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALtOpm3gS6HxAAIxr/gC3q7ckV641BdFWXdQ== --osd-uuid 0f7bd8d5-fbfd-4eeb-8fe9-6c34eed71899 2024-09-18T11:05:48.267 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.262+0000 7f3d393d1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:48.274 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.270+0000 7f3d393d1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:48.276 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.270+0000 7f3d393d1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.270+0000 7f3d393d1540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:05:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.270+0000 7f3d393d1540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:05:48.278 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.270+0000 7f3d393d1540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:05:48.366 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:48.358+0000 7f3d393d1540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T11:05:55.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T11:05:55.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:05:55.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T11:05:55.637 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T11:05:55.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:05:56.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T11:05:56.174 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T11:05:56.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:05:56.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:05:56.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:05:56.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:05:56.208 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:56.201+0000 7f8147b7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:56.232 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:56.225+0000 7f8147b7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:56.233 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:56.225+0000 7f8147b7a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:05:56.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T11:05:56.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:05:56.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T11:05:56.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:05:56.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:05:56.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:56.591 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:05:56.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:05:56.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:56.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:05:56.789 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:56.781+0000 7f8147b7a540 -1 Falling back to public interface 2024-09-18T11:05:57.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:58.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:58.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:58.017 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:05:58.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:05:58.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:58.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:05:58.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:05:59.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:05:59.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:05:59.425 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:05:59.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:05:59.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:05:59.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:05:59.714 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:05:59.705+0000 7f8147b7a540 -1 osd.4 0 log_to_monitors true 2024-09-18T11:05:59.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:00.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:00.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:00.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:06:00.928 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:06:00.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:00.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:01.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:02.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:02.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:02.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:06:02.899 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:06:02.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:02.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:03.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:04.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:04.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:04.335 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:06:04.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:06:04.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:04.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:04.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:05.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:05.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:05.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:06:05.936 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:06:05.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:05.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:06.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:07.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:07.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:07.466 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:06:07.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:06:07.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:07.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:07.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:09.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:09.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:09.002 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:06:09.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:06:09.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:09.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:09.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:10.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:10.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:10.588 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:06:10.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:06:10.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:10.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:11.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:12.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:12.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:12.042 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:06:12.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:06:12.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:12.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:12.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:13.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:13.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:13.618 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:06:13.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:06:13.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:13.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:15.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:15.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:15.282 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:06:15.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:06:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:15.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:16.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:16.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:16.953 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:06:16.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:06:16.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:16.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:18.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:19.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:19.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:19.027 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:06:19.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:06:19.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:19.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:19.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:20.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:20.810 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:06:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:06:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:21.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:22.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:22.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:22.415 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:06:22.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:06:22.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:22.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:23.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:24.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:24.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:24.575 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:06:24.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:06:24.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:24.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:25.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:26.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:26.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:26.048 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:06:26.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:06:26.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:26.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:06:26.699 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/145710,v1:127.0.0.1:6835/145710] [v2:127.0.0.1:6836/145710,v1:127.0.0.1:6837/145710] exists,up 0f7bd8d5-fbfd-4eeb-8fe9-6c34eed71899 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:06:26.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:06:26.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:06:26.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:06:26.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:06:26.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:06:26.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:06:26.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:06:26.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:06:26.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:06:26.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-18T11:06:26.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:06:26.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46b121a1-b070-4137-8d15-2428f46d81bb 2024-09-18T11:06:26.708 INFO:tasks.workunit.client.0.smithi174.stdout:add osd5 46b121a1-b070-4137-8d15-2428f46d81bb 2024-09-18T11:06:26.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 46b121a1-b070-4137-8d15-2428f46d81bb' 2024-09-18T11:06:26.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:06:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAytOpmSbyVKxAAXpgbksTQySG65VGRqY1RMg== 2024-09-18T11:06:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAytOpmSbyVKxAAXpgbksTQySG65VGRqY1RMg=="}' 2024-09-18T11:06:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46b121a1-b070-4137-8d15-2428f46d81bb -i td/test-erasure-eio/5/new.json 2024-09-18T11:06:27.588 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:06:27.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-18T11:06:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAytOpmSbyVKxAAXpgbksTQySG65VGRqY1RMg== --osd-uuid 46b121a1-b070-4137-8d15-2428f46d81bb 2024-09-18T11:06:27.632 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.625+0000 7fbbcb9b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:06:27.636 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.629+0000 7fbbcb9b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:06:27.639 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.633+0000 7fbbcb9b9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:06:27.639 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.633+0000 7fbbcb9b9540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:06:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.633+0000 7fbbcb9b9540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:06:27.640 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.633+0000 7fbbcb9b9540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:06:27.756 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:27.749+0000 7fbbcb9b9540 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-18T11:06:34.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-18T11:06:34.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:06:34.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T11:06:34.701 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd5 key to auth repository 2024-09-18T11:06:34.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:06:35.247 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.5 2024-09-18T11:06:35.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T11:06:35.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:06:35.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:06:35.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:06:35.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:06:35.281 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:35.273+0000 7f066b279540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:06:35.306 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:35.301+0000 7f066b279540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:06:35.308 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:35.301+0000 7f066b279540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:35.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:06:35.671 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:06:35.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:35.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:35.861 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:35.853+0000 7f066b279540 -1 Falling back to public interface 2024-09-18T11:06:36.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:37.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:37.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:37.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:06:37.104 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:06:37.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:37.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:37.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:38.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:38.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:38.537 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:06:38.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:06:38.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:38.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:38.695 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:06:38.689+0000 7f066b279540 -1 osd.5 0 log_to_monitors true 2024-09-18T11:06:39.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:40.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:40.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:40.039 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:06:40.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:06:40.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:40.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:40.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:41.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:41.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:41.477 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:06:41.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:06:41.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:41.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:41.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:42.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:42.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:42.935 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:06:42.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:06:42.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:42.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:43.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:44.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:44.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:44.592 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:06:44.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:06:44.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:44.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:45.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:46.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:46.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:46.054 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:06:46.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:06:46.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:46.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:46.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:47.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:47.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:47.556 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:06:47.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:06:47.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:47.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:48.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:49.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:49.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:49.005 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:06:49.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:06:49.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:49.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:49.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:50.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:50.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:50.608 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:06:50.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:06:50.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:50.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:51.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:52.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:52.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:52.061 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:06:52.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:06:52.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:52.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:52.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:53.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:53.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:53.542 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:06:53.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:06:53.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:53.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:54.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:55.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:55.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:55.099 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:06:55.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:06:55.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:55.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:55.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:56.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:56.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:56.532 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:06:56.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:06:56.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:56.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:57.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:06:58.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:06:58.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:06:58.884 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:06:58.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:06:58.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:06:58.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:06:59.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:00.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:00.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:00.722 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:07:00.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:07:00.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:00.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:07:01.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:02.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:02.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:02.372 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:07:02.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:07:02.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:02.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:07:03.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:04.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:04.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:04.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:07:04.146 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:07:04.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:04.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:07:04.581 INFO:tasks.workunit.client.0.smithi174.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/147511,v1:127.0.0.1:6843/147511] [v2:127.0.0.1:6844/147511,v1:127.0.0.1:6845/147511] exists,up 46b121a1-b070-4137-8d15-2428f46d81bb 2024-09-18T11:07:04.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:07:04.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:07:04.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:07:04.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-18T11:07:04.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:07:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:07:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-18T11:07:04.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:07:04.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63bb54ff-ae6a-415b-96ed-0c302bd05b22 2024-09-18T11:07:04.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 63bb54ff-ae6a-415b-96ed-0c302bd05b22' 2024-09-18T11:07:04.589 INFO:tasks.workunit.client.0.smithi174.stdout:add osd6 63bb54ff-ae6a-415b-96ed-0c302bd05b22 2024-09-18T11:07:04.590 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:07:04.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYtOpmY+luJBAA+bdqPA3nPBCoDaTxYv136Q== 2024-09-18T11:07:04.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYtOpmY+luJBAA+bdqPA3nPBCoDaTxYv136Q=="}' 2024-09-18T11:07:04.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63bb54ff-ae6a-415b-96ed-0c302bd05b22 -i td/test-erasure-eio/6/new.json 2024-09-18T11:07:05.311 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:07:05.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-18T11:07:05.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYtOpmY+luJBAA+bdqPA3nPBCoDaTxYv136Q== --osd-uuid 63bb54ff-ae6a-415b-96ed-0c302bd05b22 2024-09-18T11:07:05.362 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.357+0000 7f8b54126540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:07:05.579 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.577+0000 7f8b54126540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:07:05.581 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.577+0000 7f8b54126540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:07:05.583 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.581+0000 7f8b54126540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:07:05.583 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.581+0000 7f8b54126540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:07:05.583 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.581+0000 7f8b54126540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:07:05.694 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:05.689+0000 7f8b54126540 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-18T11:07:12.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-18T11:07:12.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:07:12.240 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd6 key to auth repository 2024-09-18T11:07:12.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T11:07:12.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:07:12.779 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.6 2024-09-18T11:07:12.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T11:07:12.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:07:12.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:07:12.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:07:12.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:07:12.813 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:12.808+0000 7f1774a38540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:07:12.866 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:12.860+0000 7f1774a38540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:07:12.867 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:12.864+0000 7f1774a38540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:07:13.166 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:07:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:13.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:13.433 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:13.428+0000 7f1774a38540 -1 Falling back to public interface 2024-09-18T11:07:13.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:14.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:14.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:14.646 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:07:14.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:07:14.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:14.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:15.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:16.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:16.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:16.055 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:07:16.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:07:16.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:16.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:16.119 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:07:16.116+0000 7f1774a38540 -1 osd.6 0 log_to_monitors true 2024-09-18T11:07:16.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:17.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:17.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:17.456 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:07:17.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:07:17.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:17.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:18.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:19.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:19.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:19.240 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:07:19.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:07:19.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:19.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:19.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:20.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:20.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:20.752 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:07:20.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:07:20.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:20.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:21.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:22.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:22.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:22.212 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:07:22.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:07:22.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:22.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:22.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:23.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:23.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:23.699 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:07:23.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:07:23.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:23.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:24.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:25.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:25.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:25.262 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:07:25.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:07:25.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:25.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:25.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:26.734 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:07:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:07:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:27.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:28.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:28.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:28.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:07:28.239 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:07:28.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:28.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:28.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:29.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:29.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:29.941 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:07:29.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:07:29.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:29.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:30.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:31.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:31.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:31.584 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:07:31.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:07:31.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:31.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:32.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:33.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:33.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:33.094 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:07:33.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:07:33.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:33.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:34.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:35.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:35.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:35.132 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:07:35.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:07:35.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:35.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:35.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:36.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:36.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:36.919 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:07:36.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:07:36.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:36.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:37.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:38.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:38.698 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:07:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:07:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:39.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:40.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:40.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:40.288 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:07:40.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:07:40.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:40.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:40.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:07:41.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:07:41.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:07:41.746 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:07:41.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:07:41.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:07:41.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:07:42.263 INFO:tasks.workunit.client.0.smithi174.stdout:osd.6 up in weight 1 up_from 58 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/149314,v1:127.0.0.1:6851/149314] [v2:127.0.0.1:6852/149314,v1:127.0.0.1:6853/149314] exists,up 63bb54ff-ae6a-415b-96ed-0c302bd05b22 2024-09-18T11:07:42.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:07:42.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:07:42.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:07:42.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:07:42.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:07:42.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:07:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:07:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:07:42.266 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:07:42.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:07:42.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:07:42.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:07:42.347 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:07:42.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:07:42.367 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:03:32.799+0000 7fd98d59d540 0 load: jerasure load: lrc 2024-09-18T11:07:42.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:07:42.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-09-18T11:07:42.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-09-18T11:07:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:07:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:07:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-18T11:07:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:07:42.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-18T11:07:42.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:07:42.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-18T11:07:42.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:07:42.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:07:44.321 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:07:44.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:07:45.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:07:45.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:07:45.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:07:45.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:07:45.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:07:45.339 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:07:45.340 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:07:45.340 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:07:45.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:07:45.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:07:45.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:07:45.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:07:45.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:07:45.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:07:45.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:07:45.552 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:07:46.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:07:46.068 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:07:46.068 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:07:46.068 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:07:46.069 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:07:46.069 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:07:46.069 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:07:46.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:07:46.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:07:46.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803831 2024-09-18T11:07:46.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803831 2024-09-18T11:07:46.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831' 2024-09-18T11:07:46.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.208 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:07:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574891 2024-09-18T11:07:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574891 2024-09-18T11:07:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-55834574891' 2024-09-18T11:07:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:07:46.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378666 2024-09-18T11:07:46.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378666 2024-09-18T11:07:46.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-81604378666' 2024-09-18T11:07:46.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.510 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:07:46.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084319 2024-09-18T11:07:46.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084319 2024-09-18T11:07:46.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-81604378666 3-120259084319' 2024-09-18T11:07:46.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.653 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:07:46.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724572 2024-09-18T11:07:46.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724572 2024-09-18T11:07:46.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-81604378666 3-120259084319 4-167503724572' 2024-09-18T11:07:46.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.808 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:07:46.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397525 2024-09-18T11:07:46.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397525 2024-09-18T11:07:46.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-81604378666 3-120259084319 4-167503724572 5-210453397525' 2024-09-18T11:07:46.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:46.955 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:07:47.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=249108103175 2024-09-18T11:07:47.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 249108103175 2024-09-18T11:07:47.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803831 1-55834574891 2-81604378666 3-120259084319 4-167503724572 5-210453397525 6-249108103175' 2024-09-18T11:07:47.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:47.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803831 2024-09-18T11:07:47.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:47.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:07:47.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803831 2024-09-18T11:07:47.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:47.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803831 2024-09-18T11:07:47.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-09-18T11:07:47.109 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803831 2024-09-18T11:07:47.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:07:47.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803831 -lt 25769803831 2024-09-18T11:07:47.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:47.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574891 2024-09-18T11:07:47.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:47.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:07:47.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574891 2024-09-18T11:07:47.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:47.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574891 2024-09-18T11:07:47.546 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574891 2024-09-18T11:07:47.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574891' 2024-09-18T11:07:47.547 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:07:47.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574891 -lt 55834574891 2024-09-18T11:07:47.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:47.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378666 2024-09-18T11:07:47.955 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:47.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:07:47.958 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:47.958 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378666 2024-09-18T11:07:47.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378666 2024-09-18T11:07:47.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378666' 2024-09-18T11:07:47.960 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378666 2024-09-18T11:07:47.960 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:07:48.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378666 -lt 81604378666 2024-09-18T11:07:48.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:48.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084319 2024-09-18T11:07:48.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:48.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:07:48.493 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:48.493 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084319 2024-09-18T11:07:48.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084319 2024-09-18T11:07:48.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084319' 2024-09-18T11:07:48.494 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084319 2024-09-18T11:07:48.495 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:07:48.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084320 -lt 120259084319 2024-09-18T11:07:48.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:48.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:48.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724572 2024-09-18T11:07:48.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:07:48.976 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724572 2024-09-18T11:07:48.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:48.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724572 2024-09-18T11:07:48.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724572' 2024-09-18T11:07:48.979 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724572 2024-09-18T11:07:48.979 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:07:49.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724572 -lt 167503724572 2024-09-18T11:07:49.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:49.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-210453397525 2024-09-18T11:07:49.402 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:49.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:07:49.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-210453397525 2024-09-18T11:07:49.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:49.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397525 2024-09-18T11:07:49.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 210453397525' 2024-09-18T11:07:49.407 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 210453397525 2024-09-18T11:07:49.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:07:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397526 -lt 210453397525 2024-09-18T11:07:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:49.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-249108103175 2024-09-18T11:07:49.828 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:49.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:07:49.831 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-249108103175 2024-09-18T11:07:49.832 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:49.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=249108103175 2024-09-18T11:07:49.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 249108103175' 2024-09-18T11:07:49.834 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 249108103175 2024-09-18T11:07:49.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:07:50.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 249108103175 -lt 249108103175 2024-09-18T11:07:50.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:07:50.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:07:50.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:07:50.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:07:50.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:07:50.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:07:50.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:07:50.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:07:50.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:07:50.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:07:50.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:07:51.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:07:51.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:07:51.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:07:51.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-18T11:07:51.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:07:51.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-18T11:07:51.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-18T11:07:51.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:07:51.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:07:51.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:07:51.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-18T11:07:52.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-18T11:07:52.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:07:52.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:07:52.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:07:52.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-18T11:07:52.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-09-18T11:07:53.345 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already out. 2024-09-18T11:07:53.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-09-18T11:07:53.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:07:53.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:07:53.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:07:53.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:07:53.359 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:07:53.360 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:07:53.360 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:07:53.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:07:53.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:07:53.361 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:07:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:07:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:07:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:07:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:07:53.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:07:54.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:54.187 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:07:54.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803835 2024-09-18T11:07:54.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803835 2024-09-18T11:07:54.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835' 2024-09-18T11:07:54.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:54.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:07:54.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574895 2024-09-18T11:07:54.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574895 2024-09-18T11:07:54.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-55834574895' 2024-09-18T11:07:54.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:54.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:07:54.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378670 2024-09-18T11:07:54.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378670 2024-09-18T11:07:54.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-81604378670' 2024-09-18T11:07:54.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:54.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:07:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084323 2024-09-18T11:07:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084323 2024-09-18T11:07:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-81604378670 3-120259084323' 2024-09-18T11:07:55.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:55.025 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:07:55.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724575 2024-09-18T11:07:55.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724575 2024-09-18T11:07:55.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-81604378670 3-120259084323 4-167503724575' 2024-09-18T11:07:55.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:55.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:07:55.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397529 2024-09-18T11:07:55.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397529 2024-09-18T11:07:55.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-81604378670 3-120259084323 4-167503724575 5-210453397529' 2024-09-18T11:07:55.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:07:55.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:07:55.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=249108103178 2024-09-18T11:07:55.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 249108103178 2024-09-18T11:07:55.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-55834574895 2-81604378670 3-120259084323 4-167503724575 5-210453397529 6-249108103178' 2024-09-18T11:07:55.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:55.453 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803835 2024-09-18T11:07:55.453 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:55.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:07:55.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803835 2024-09-18T11:07:55.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:55.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803835 2024-09-18T11:07:55.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-18T11:07:55.458 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803835 2024-09-18T11:07:55.458 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:07:55.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803835 -lt 25769803835 2024-09-18T11:07:55.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:55.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574895 2024-09-18T11:07:55.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:55.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:07:55.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574895 2024-09-18T11:07:55.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:55.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574895 2024-09-18T11:07:55.877 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574895 2024-09-18T11:07:55.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-18T11:07:55.877 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:07:56.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574895 -lt 55834574895 2024-09-18T11:07:56.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:56.300 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378670 2024-09-18T11:07:56.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:56.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:07:56.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378670 2024-09-18T11:07:56.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:56.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378670 2024-09-18T11:07:56.306 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378670 2024-09-18T11:07:56.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378670' 2024-09-18T11:07:56.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:07:56.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378670 -lt 81604378670 2024-09-18T11:07:56.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:56.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:56.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084323 2024-09-18T11:07:56.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:07:56.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084323 2024-09-18T11:07:56.818 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:56.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084323 2024-09-18T11:07:56.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084323' 2024-09-18T11:07:56.820 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084323 2024-09-18T11:07:56.821 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:07:57.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084323 -lt 120259084323 2024-09-18T11:07:57.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:57.234 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724575 2024-09-18T11:07:57.234 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:57.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:07:57.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724575 2024-09-18T11:07:57.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:57.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724575 2024-09-18T11:07:57.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724575' 2024-09-18T11:07:57.239 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724575 2024-09-18T11:07:57.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:07:57.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724576 -lt 167503724575 2024-09-18T11:07:57.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:57.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-210453397529 2024-09-18T11:07:57.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:57.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:07:57.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-210453397529 2024-09-18T11:07:57.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:57.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397529 2024-09-18T11:07:57.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 210453397529' 2024-09-18T11:07:57.672 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 210453397529 2024-09-18T11:07:57.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:07:58.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397529 -lt 210453397529 2024-09-18T11:07:58.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:07:58.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-249108103178 2024-09-18T11:07:58.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:07:58.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:07:58.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-249108103178 2024-09-18T11:07:58.100 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:07:58.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=249108103178 2024-09-18T11:07:58.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 249108103178' 2024-09-18T11:07:58.103 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 249108103178 2024-09-18T11:07:58.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:07:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 249108103179 -lt 249108103178 2024-09-18T11:07:58.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:07:58.521 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:07:58.521 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:07:59.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:07:59.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:07:59.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:07:59.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:07:59.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:07:59.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:07:59.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:07:59.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:07:59.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:07:59.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:07:59.483 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:07:59.483 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:08:00.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:08:00.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-18T11:08:00.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:08:00.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-18T11:08:00.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:08:00.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:08:00.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:08:00.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:08:00.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:08:00.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:08:00.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:08:00.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:08:00.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:08:00.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:08:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:08:00.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:08:00.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:08:00.498 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:08:01.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:08:01.025 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:08:01.025 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:08:01.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1535 2024-09-18T11:08:01.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1535 '!=' null 2024-09-18T11:08:01.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T11:08:01.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:08:01.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:08:01.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:08:01.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:08:01.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:08:01.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:08:01.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:08:01.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:08:01.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:08:01.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:08:02.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:08:02.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:08:02.028 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:08:02.028 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:08:02.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:08:02.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:08:02.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:08:02.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-18T11:08:02.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:08:02.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:08:02.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-18T11:08:02.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:08:02.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:08:02.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-18T11:08:02.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:08:02.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:08:02.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:08:02.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:08:02.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:08:03.413 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:08:03.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:08:04.176 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:08:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:08:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:08:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:08:04.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:08:04.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:08:04.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:08:04.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:08:04.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:08:04.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:08:04.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:08:04.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:08:04.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:08:04.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:08:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:08:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:08:04.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:08:04.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:08:04.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:08:04.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:08:04.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:08:04.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:08:04.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:08:04.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:08:04.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:08:04.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:04.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:04.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:08:04.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:08:04.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:08:04.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:08:04.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:08:04.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:08:04.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:08:04.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:08:04.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:08:04.559 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:08:04.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:08:04.562 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:08:04.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:08:04.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:08:04.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:08:04.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:08:04.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:08:04.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:08:04.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:08:04.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:08:04.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:08:04.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:08:04.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:08:04.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:04.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:04.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:08:04.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:08:04.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:08:04.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:08:04.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:08:04.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:04.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:04.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:08:04.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:08:04.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:08:04.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:08:04.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:08:04.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:08:04.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:08:04.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:08:04.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:08:04.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:04.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:04.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:08:04.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:08:05.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:08:05.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:08:05.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:08:05.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:08:05.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:08:05.256 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:08:05.256 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:08:05.256 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:08:05.256 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:08:05.257 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:08:05.257 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:05.257 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:05.258 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:08:05.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:08:05.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:08:05.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:08:05.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:08:05.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:08:05.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:08:05.344 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:08:05.344 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:08:05.344 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:08:05.345 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:08:05.345 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:08:05.345 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:05.345 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:05.346 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:08:05.346 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:08:05.347 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:08:05.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:08:06.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:08:06.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:08:06.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:08:06.342 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:08:06.342 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:06.342 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:06.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:08:06.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:08:06.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:08:06.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:08:06.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:08:06.721 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:08:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:08:07.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:08:07.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:08:07.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:08:07.733 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:08:07.733 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:07.733 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:07.733 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:08:07.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:08:07.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:08:07.811 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:08:07.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:08:07.823 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:08:04.988+0000 7f387c123a00 0 load: jerasure load: lrc 2024-09-18T11:08:07.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-09-18T11:08:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-09-18T11:08:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-09-18T11:08:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:08:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:08:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-09-18T11:08:07.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-18T11:08:07.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:08:07.825 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-18T11:08:07.825 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-18T11:08:07.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:08:07.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:08:07.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:08:07.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:08:07.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:08:07.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:08:07.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:08:07.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:08:07.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:08:07.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:08:07.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:08:07.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:08:07.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:08:07.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:08:07.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:08:07.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:08:07.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:08:07.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:08:07.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:08:07.834 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:08:07.834 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:07.834 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:07.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:08:07.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:08:07.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:08:07.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:08:07.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:08:07.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:08:07.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:08:07.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:08:07.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:08:07.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:08:07.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:08:07.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:08:07.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:08:07.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:08:07.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:08:07.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=771b45c1-7a65-4284-bdb0-37817d035076 2024-09-18T11:08:07.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 771b45c1-7a65-4284-bdb0-37817d035076' 2024-09-18T11:08:07.839 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 771b45c1-7a65-4284-bdb0-37817d035076 2024-09-18T11:08:07.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:08:07.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXtOpmI97fMhAASg9RcjZq+Vx+uX0/P7XEUw== 2024-09-18T11:08:07.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXtOpmI97fMhAASg9RcjZq+Vx+uX0/P7XEUw=="}' 2024-09-18T11:08:07.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 771b45c1-7a65-4284-bdb0-37817d035076 -i td/test-erasure-eio/0/new.json 2024-09-18T11:08:08.216 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:08:08.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:08:08.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXtOpmI97fMhAASg9RcjZq+Vx+uX0/P7XEUw== --osd-uuid 771b45c1-7a65-4284-bdb0-37817d035076 2024-09-18T11:08:08.266 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.264+0000 7f063322e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:08.269 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.264+0000 7f063322e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:08.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.268+0000 7f063322e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:08.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.268+0000 7f063322e540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:08:08.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.268+0000 7f063322e540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:08:08.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.268+0000 7f063322e540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:08:08.339 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:08.336+0000 7f063322e540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:08:14.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:08:14.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:08:14.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:08:14.058 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:08:14.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:08:14.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:08:14.605 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:08:14.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:08:14.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:08:14.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:08:14.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:08:14.640 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:14.635+0000 7fd0b9749540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:14.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:14.671+0000 7fd0b9749540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:14.679 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:14.675+0000 7fd0b9749540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:15.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:08:15.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:08:15.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:08:15.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:08:15.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:08:15.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:15.130 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:08:15.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:08:15.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:15.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:15.239 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:15.235+0000 7fd0b9749540 -1 Falling back to public interface 2024-09-18T11:08:15.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:16.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:16.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:16.616 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:08:16.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:08:16.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:16.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:17.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:17.793 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:17.787+0000 7fd0b9749540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:08:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:18.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:08:18.030 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:08:18.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:18.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:18.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:19.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:19.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:19.496 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:08:19.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:08:19.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:19.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:19.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:20.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:20.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:20.915 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:08:20.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:08:20.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:20.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:21.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:22.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:22.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:22.349 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:08:22.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:08:22.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:22.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:22.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:23.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:23.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:23.864 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:08:23.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:08:23.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:23.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:24.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:25.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:25.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:25.310 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:08:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:08:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:25.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:26.860 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:08:26.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:08:26.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:26.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:27.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:28.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:28.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:28.297 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:08:28.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:08:28.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:28.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:28.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:29.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:29.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:29.837 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:08:29.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:08:29.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:29.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:30.478 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:31.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:31.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:31.480 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:08:31.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:08:31.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:31.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:32.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:33.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:33.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:33.103 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:08:33.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:08:33.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:33.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:34.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:35.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:35.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:35.039 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:08:35.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:08:35.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:35.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:35.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:36.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:36.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:36.516 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:08:36.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:08:36.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:36.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:37.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:38.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:38.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:38.051 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:08:38.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:08:38.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:38.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:38.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:39.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:39.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:39.561 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:08:39.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:08:39.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:39.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:40.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:41.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:41.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:41.541 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:08:41.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:08:41.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:41.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:08:42.103 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/153028,v1:127.0.0.1:6803/153028] [v2:127.0.0.1:6804/153028,v1:127.0.0.1:6805/153028] exists,up 771b45c1-7a65-4284-bdb0-37817d035076 2024-09-18T11:08:42.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:08:42.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:08:42.108 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:08:42.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:08:42.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:08:42.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:08:42.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:08:42.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:08:42.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:08:42.111 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:08:42.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=020128b4-2ca6-4180-80c4-9d322e016a60 2024-09-18T11:08:42.112 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 020128b4-2ca6-4180-80c4-9d322e016a60 2024-09-18T11:08:42.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 020128b4-2ca6-4180-80c4-9d322e016a60' 2024-09-18T11:08:42.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:08:42.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6tOpmqUQUCBAABi3tSihXtsjbx9azVkWSpw== 2024-09-18T11:08:42.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6tOpmqUQUCBAABi3tSihXtsjbx9azVkWSpw=="}' 2024-09-18T11:08:42.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 020128b4-2ca6-4180-80c4-9d322e016a60 -i td/test-erasure-eio/1/new.json 2024-09-18T11:08:42.725 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:08:42.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:08:42.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6tOpmqUQUCBAABi3tSihXtsjbx9azVkWSpw== --osd-uuid 020128b4-2ca6-4180-80c4-9d322e016a60 2024-09-18T11:08:42.768 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.763+0000 7f59e294c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:42.781 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.775+0000 7f59e294c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:42.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.779+0000 7f59e294c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:42.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.779+0000 7f59e294c540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:08:42.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.779+0000 7f59e294c540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:08:42.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.779+0000 7f59e294c540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:08:42.836 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:42.831+0000 7f59e294c540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:08:48.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:08:48.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:08:48.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:08:48.805 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:08:48.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:08:49.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:08:49.347 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:08:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:08:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:08:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:08:49.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:08:49.381 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:49.375+0000 7f6cd2d06540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:49.385 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:49.379+0000 7f6cd2d06540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:49.385 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:49.379+0000 7f6cd2d06540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:08:49.827 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:08:49.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:49.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:49.953 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:49.947+0000 7f6cd2d06540 -1 Falling back to public interface 2024-09-18T11:08:50.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:51.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:51.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:51.269 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:08:51.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:08:51.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:51.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:51.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:52.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:52.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:52.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:08:52.712 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:08:52.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:52.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:52.717 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:08:52.711+0000 7f6cd2d06540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:08:53.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:54.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:54.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:54.236 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:08:54.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:08:54.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:54.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:54.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:55.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:55.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:55.686 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:08:55.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:08:55.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:55.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:56.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:57.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:57.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:57.107 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:08:57.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:08:57.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:57.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:57.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:08:58.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:08:58.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:08:58.698 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:08:58.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:08:58.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:08:58.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:08:59.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:00.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:00.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:00.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:09:00.205 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:09:00.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:00.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:00.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:01.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:01.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:01.642 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:09:01.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:09:01.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:01.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:02.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:03.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:03.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:03.076 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:09:03.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:09:03.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:03.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:04.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:04.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:04.579 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:09:04.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:09:04.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:04.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:05.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:06.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:09:06.028 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:09:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:06.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:07.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:07.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:07.591 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:09:07.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:09:07.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:07.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:08.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:09.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:09.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:09.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:09:09.523 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:09:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:10.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:11.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:11.390 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:11.390 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:09:11.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:09:11.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:11.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:12.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:13.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:13.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:13.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:09:13.289 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:09:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:14.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:15.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:15.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:15.010 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:09:15.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:09:15.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:15.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:15.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:16.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:16.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:16.476 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:09:16.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:09:16.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:16.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:17.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:18.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:18.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:18.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:09:18.033 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:09:18.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:18.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:09:18.603 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/154801,v1:127.0.0.1:6811/154801] [v2:127.0.0.1:6812/154801,v1:127.0.0.1:6813/154801] exists,up 020128b4-2ca6-4180-80c4-9d322e016a60 2024-09-18T11:09:18.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:09:18.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:09:18.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:09:18.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:09:18.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:09:18.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:09:18.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:09:18.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:09:18.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:09:18.607 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:09:18.608 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:09:18.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:09:18.610 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:09:18.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a9588da-c970-4685-8ba8-1a211921a7bc 2024-09-18T11:09:18.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2a9588da-c970-4685-8ba8-1a211921a7bc' 2024-09-18T11:09:18.611 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 2a9588da-c970-4685-8ba8-1a211921a7bc 2024-09-18T11:09:18.612 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:09:18.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDetOpmxGKTJRAAABuRKEP9kuze0FYXYtkATg== 2024-09-18T11:09:18.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDetOpmxGKTJRAAABuRKEP9kuze0FYXYtkATg=="}' 2024-09-18T11:09:18.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a9588da-c970-4685-8ba8-1a211921a7bc -i td/test-erasure-eio/2/new.json 2024-09-18T11:09:19.201 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:09:19.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:09:19.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDetOpmxGKTJRAAABuRKEP9kuze0FYXYtkATg== --osd-uuid 2a9588da-c970-4685-8ba8-1a211921a7bc 2024-09-18T11:09:19.252 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.247+0000 7ff2158a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:19.255 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.251+0000 7ff2158a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:19.257 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.251+0000 7ff2158a3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:19.258 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.255+0000 7ff2158a3540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:09:19.258 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.255+0000 7ff2158a3540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:09:19.258 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.255+0000 7ff2158a3540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:09:19.357 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:19.351+0000 7ff2158a3540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:09:25.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:09:25.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:09:25.908 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:09:25.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:09:25.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:09:26.436 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:09:26.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:09:26.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:09:26.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:09:26.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:09:26.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:09:26.471 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:26.466+0000 7fedb5a93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:26.473 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:26.466+0000 7fedb5a93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:26.474 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:26.470+0000 7fedb5a93540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:09:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:09:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:09:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:09:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:09:26.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:26.861 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:09:26.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:09:26.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:26.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:27.024 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:27.018+0000 7fedb5a93540 -1 Falling back to public interface 2024-09-18T11:09:27.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:28.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:28.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:28.236 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:09:28.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:09:28.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:28.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:28.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:29.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:29.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:29.672 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:09:29.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:09:29.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:29.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:29.748 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:29.742+0000 7fedb5a93540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:09:30.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:31.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:31.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:31.081 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:09:31.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:09:31.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:31.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:31.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:32.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:32.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:32.538 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:09:32.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:09:32.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:32.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:32.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:33.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:33.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:33.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:09:33.979 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:09:33.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:33.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:34.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:35.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:35.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:35.583 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:09:35.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:09:35.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:35.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:36.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:37.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:37.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:37.038 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:09:37.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:09:37.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:37.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:37.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:38.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:38.552 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:09:38.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:09:38.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:38.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:39.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:40.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:40.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:40.040 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:09:40.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:09:40.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:40.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:40.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:41.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:41.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:41.558 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:09:41.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:09:41.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:41.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:42.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:43.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:43.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:43.006 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:09:43.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:09:43.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:43.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:43.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:44.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:44.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:44.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:09:44.501 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:09:44.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:44.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:44.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:45.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:45.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:45.927 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:09:45.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:09:45.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:45.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:46.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:47.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:47.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:47.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:09:47.546 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:09:47.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:47.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:48.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:49.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:49.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:49.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:09:49.424 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:09:49.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:49.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:49.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:50.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:50.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:50.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:09:50.968 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:09:50.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:50.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:52.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:53.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:53.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:53.152 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:09:53.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:09:53.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:53.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:53.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:09:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:09:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:09:54.611 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:09:54.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:09:54.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:09:54.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:09:55.026 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/156603,v1:127.0.0.1:6819/156603] [v2:127.0.0.1:6820/156603,v1:127.0.0.1:6821/156603] exists,up 2a9588da-c970-4685-8ba8-1a211921a7bc 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:09:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:09:55.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:09:55.032 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:09:55.032 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:09:55.032 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:09:55.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:09:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:09:55.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:09:55.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63ba9332-a0fc-464e-8048-9e8669e1ac79 2024-09-18T11:09:55.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 63ba9332-a0fc-464e-8048-9e8669e1ac79' 2024-09-18T11:09:55.035 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 63ba9332-a0fc-464e-8048-9e8669e1ac79 2024-09-18T11:09:55.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:09:55.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADtepmfVAxAxAA3to9n2cFeTqYUgqyCXOgXQ== 2024-09-18T11:09:55.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADtepmfVAxAxAA3to9n2cFeTqYUgqyCXOgXQ=="}' 2024-09-18T11:09:55.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63ba9332-a0fc-464e-8048-9e8669e1ac79 -i td/test-erasure-eio/3/new.json 2024-09-18T11:09:55.673 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:09:55.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:09:55.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADtepmfVAxAxAA3to9n2cFeTqYUgqyCXOgXQ== --osd-uuid 63ba9332-a0fc-464e-8048-9e8669e1ac79 2024-09-18T11:09:55.720 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.714+0000 7f895467d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:55.723 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.718+0000 7f895467d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:55.725 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.718+0000 7f895467d540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:09:55.726 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.722+0000 7f895467d540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:09:55.726 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.722+0000 7f895467d540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:09:55.726 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.722+0000 7f895467d540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:09:55.807 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:09:55.802+0000 7f895467d540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:10:02.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:10:02.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:10:02.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:10:02.097 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:10:02.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:10:02.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:10:02.698 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:10:02.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:10:02.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:10:02.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:10:02.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:10:02.732 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:02.726+0000 7f6ce47ee540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:02.746 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:02.738+0000 7f6ce47ee540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:02.747 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:02.742+0000 7f6ce47ee540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:10:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:10:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:10:03.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:10:03.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:10:03.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:03.085 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:10:03.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:10:03.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:03.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:03.317 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:03.310+0000 7f6ce47ee540 -1 Falling back to public interface 2024-09-18T11:10:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:04.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:04.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:04.516 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:10:04.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:10:04.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:04.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:04.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:05.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:05.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:05.924 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:10:05.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:10:05.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:05.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:06.134 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:06.130+0000 7f6ce47ee540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:10:06.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:07.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:07.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:07.350 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:10:07.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:10:07.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:07.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:08.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:09.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:09.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:09.083 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:10:09.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:10:09.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:09.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:09.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:10.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:10.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:10.525 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:10:10.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:10:10.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:10.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:11.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:12.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:12.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:12.159 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:10:12.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:10:12.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:12.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:12.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:13.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:13.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:13.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:10:13.726 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:10:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:13.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:14.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:15.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:15.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:15.234 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:10:15.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:10:15.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:15.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:15.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:16.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:16.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:16.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:10:16.640 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:10:16.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:16.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:17.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:18.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:18.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:18.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:10:18.221 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:10:18.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:18.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:18.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:19.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:19.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:19.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:10:19.716 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:10:19.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:19.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:20.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:21.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:21.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:21.663 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:10:21.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:10:21.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:21.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:22.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:23.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:23.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:23.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:10:23.435 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:10:23.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:23.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:24.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:25.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:25.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:25.027 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:10:25.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:10:25.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:25.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:25.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:26.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:26.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:26.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:10:26.805 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:10:26.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:26.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:27.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:28.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:28.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:28.992 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:10:28.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:10:28.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:28.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:29.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:30.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:30.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:30.685 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:10:30.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:10:30.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:30.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:10:31.220 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/158405,v1:127.0.0.1:6827/158405] [v2:127.0.0.1:6828/158405,v1:127.0.0.1:6829/158405] exists,up 63ba9332-a0fc-464e-8048-9e8669e1ac79 2024-09-18T11:10:31.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:10:31.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T11:10:31.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:10:31.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:10:31.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:10:31.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:10:31.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:10:31.229 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:10:31.233 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:10:31.233 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:10:31.233 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:10:31.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:10:31.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:10:31.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:10:31.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:10:31.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:10:31.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:10:31.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T11:10:31.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:10:31.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4e59e4b5-1bab-4f00-8f82-a64dffa28c15 2024-09-18T11:10:31.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 4e59e4b5-1bab-4f00-8f82-a64dffa28c15' 2024-09-18T11:10:31.236 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 4e59e4b5-1bab-4f00-8f82-a64dffa28c15 2024-09-18T11:10:31.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:10:31.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAntepmcLWUDhAAW4FQHJ7NL0XzxpG1XUhQrA== 2024-09-18T11:10:31.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAntepmcLWUDhAAW4FQHJ7NL0XzxpG1XUhQrA=="}' 2024-09-18T11:10:31.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4e59e4b5-1bab-4f00-8f82-a64dffa28c15 -i td/test-erasure-eio/4/new.json 2024-09-18T11:10:31.960 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:10:31.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T11:10:31.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAntepmcLWUDhAAW4FQHJ7NL0XzxpG1XUhQrA== --osd-uuid 4e59e4b5-1bab-4f00-8f82-a64dffa28c15 2024-09-18T11:10:32.018 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.013+0000 7f7c396d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:32.021 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.013+0000 7f7c396d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:32.023 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.017+0000 7f7c396d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:32.024 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.017+0000 7f7c396d3540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:10:32.024 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.017+0000 7f7c396d3540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:10:32.025 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.017+0000 7f7c396d3540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:10:32.098 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:32.093+0000 7f7c396d3540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T11:10:39.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T11:10:39.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:10:39.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T11:10:39.897 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T11:10:39.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:10:40.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T11:10:40.442 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T11:10:40.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:10:40.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:10:40.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:10:40.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:10:40.476 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:40.469+0000 7f68ab556540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:40.499 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:40.493+0000 7f68ab556540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:40.502 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:40.493+0000 7f68ab556540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:10:40.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:10:40.927 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:10:40.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:40.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:41.073 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:41.065+0000 7f68ab556540 -1 Falling back to public interface 2024-09-18T11:10:41.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:42.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:42.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:42.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:10:42.341 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:10:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:42.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:42.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:10:43.810 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:10:43.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:43.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:43.851 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:10:43.845+0000 7f68ab556540 -1 osd.4 0 log_to_monitors true 2024-09-18T11:10:44.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:45.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:45.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:45.373 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:10:45.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:10:45.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:45.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:46.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:47.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:47.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:47.081 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:10:47.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:10:47.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:47.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:47.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:48.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:48.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:48.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:10:48.522 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:10:48.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:48.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:49.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:50.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:50.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:50.054 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:10:50.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:10:50.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:50.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:50.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:51.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:51.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:51.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:10:51.486 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:10:51.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:51.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:52.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:53.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:10:53.042 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:10:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:53.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:54.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:54.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:54.494 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:10:54.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:10:54.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:54.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:54.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:55.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:55.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:55.975 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:10:55.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:10:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:56.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:57.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:57.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:57.403 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:10:57.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:10:57.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:57.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:10:58.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:10:59.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:10:59.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:10:59.398 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:10:59.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:10:59.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:10:59.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:11:00.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:01.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:01.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:01.129 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:11:01.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:11:01.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:01.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:11:01.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:02.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:02.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:02.844 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:11:02.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:11:02.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:02.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:11:04.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:05.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:05.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:05.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:11:05.030 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:11:05.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:05.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:11:06.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:07.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:07.053 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:11:07.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:11:07.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:07.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:11:07.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:08.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:08.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:08.713 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:11:08.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:11:08.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:08.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:11:09.326 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/160181,v1:127.0.0.1:6835/160181] [v2:127.0.0.1:6836/160181,v1:127.0.0.1:6837/160181] exists,up 4e59e4b5-1bab-4f00-8f82-a64dffa28c15 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:11:09.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:11:09.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:11:09.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:11:09.330 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:11:09.330 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:11:09.330 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:11:09.331 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:11:09.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:11:09.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:11:09.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:11:09.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-18T11:11:09.333 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:11:09.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5528560-6e5b-42e9-a855-839008a5663f 2024-09-18T11:11:09.334 INFO:tasks.workunit.client.0.smithi174.stdout:add osd5 a5528560-6e5b-42e9-a855-839008a5663f 2024-09-18T11:11:09.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a5528560-6e5b-42e9-a855-839008a5663f' 2024-09-18T11:11:09.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:11:09.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNtepmgMQpFRAA9ahKGfRGtiH3GQwLNR7USQ== 2024-09-18T11:11:09.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNtepmgMQpFRAA9ahKGfRGtiH3GQwLNR7USQ=="}' 2024-09-18T11:11:09.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5528560-6e5b-42e9-a855-839008a5663f -i td/test-erasure-eio/5/new.json 2024-09-18T11:11:10.649 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:11:10.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-18T11:11:10.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNtepmgMQpFRAA9ahKGfRGtiH3GQwLNR7USQ== --osd-uuid a5528560-6e5b-42e9-a855-839008a5663f 2024-09-18T11:11:10.707 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.701+0000 7fa5f0391540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.717+0000 7fa5f0391540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:10.726 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.717+0000 7fa5f0391540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:10.727 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.721+0000 7fa5f0391540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:11:10.727 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.721+0000 7fa5f0391540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:11:10.727 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.721+0000 7fa5f0391540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:11:10.826 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:10.817+0000 7fa5f0391540 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-18T11:11:17.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-18T11:11:17.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:11:17.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T11:11:17.833 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd5 key to auth repository 2024-09-18T11:11:17.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:11:18.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T11:11:18.386 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.5 2024-09-18T11:11:18.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:11:18.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:11:18.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:11:18.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:11:18.420 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:18.413+0000 7f66fcbb9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:18.523 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:18.517+0000 7f66fcbb9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:18.525 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:18.517+0000 7f66fcbb9540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:18.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T11:11:18.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:11:18.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T11:11:18.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:11:18.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:11:18.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:18.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:11:18.837 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:11:18.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:18.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:19.085 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:19.077+0000 7f66fcbb9540 -1 Falling back to public interface 2024-09-18T11:11:19.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:20.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:20.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:20.270 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:11:20.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:11:20.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:20.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:20.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:21.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:21.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:21.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:11:21.696 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:11:21.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:21.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:21.823 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:21.817+0000 7f66fcbb9540 -1 osd.5 0 log_to_monitors true 2024-09-18T11:11:22.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:23.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:23.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:23.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:11:23.160 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:11:23.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:23.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:23.654 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:24.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:24.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:24.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:11:24.656 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:11:24.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:24.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:25.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:26.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:26.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:26.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:11:26.123 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:11:26.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:26.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:26.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:27.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:27.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:27.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:11:27.626 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:11:27.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:27.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:28.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:29.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:29.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:29.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:11:29.090 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:11:29.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:29.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:29.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:30.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:30.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:30.651 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:11:30.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:11:30.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:30.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:31.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:32.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:32.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:32.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:11:32.083 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:11:32.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:32.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:32.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:33.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:33.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:33.658 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:11:33.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:11:33.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:33.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:34.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:35.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:35.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:35.112 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:11:35.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:11:35.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:35.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:35.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:36.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:36.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:36.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:11:36.779 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:11:36.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:36.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:37.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:38.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:38.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:38.684 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:11:38.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:11:38.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:38.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:39.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:11:40.604 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:11:40.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:40.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:41.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:42.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:42.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:42.090 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:11:42.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:11:42.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:42.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:42.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:43.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:43.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:43.559 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:11:43.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:11:43.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:43.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:44.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:45.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:45.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:45.546 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:11:45.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:11:45.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:45.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:46.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:47.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:47.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:47.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:11:47.268 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:11:47.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:47.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:11:47.751 INFO:tasks.workunit.client.0.smithi174.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/161975,v1:127.0.0.1:6843/161975] [v2:127.0.0.1:6844/161975,v1:127.0.0.1:6845/161975] exists,up a5528560-6e5b-42e9-a855-839008a5663f 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:11:47.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:11:47.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:11:47.756 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:11:47.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:11:47.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:11:47.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:11:47.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-18T11:11:47.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:11:47.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a855820-3e54-42f9-ac31-60b227e41e8b 2024-09-18T11:11:47.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 2a855820-3e54-42f9-ac31-60b227e41e8b' 2024-09-18T11:11:47.759 INFO:tasks.workunit.client.0.smithi174.stdout:add osd6 2a855820-3e54-42f9-ac31-60b227e41e8b 2024-09-18T11:11:47.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:11:47.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBztepmdXlOLhAAZZOA2KbGfvi9sgIoNgUjPA== 2024-09-18T11:11:47.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBztepmdXlOLhAAZZOA2KbGfvi9sgIoNgUjPA=="}' 2024-09-18T11:11:47.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a855820-3e54-42f9-ac31-60b227e41e8b -i td/test-erasure-eio/6/new.json 2024-09-18T11:11:48.437 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:11:48.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-18T11:11:48.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBztepmdXlOLhAAZZOA2KbGfvi9sgIoNgUjPA== --osd-uuid 2a855820-3e54-42f9-ac31-60b227e41e8b 2024-09-18T11:11:48.494 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.488+0000 7f40674e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:48.560 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.556+0000 7f40674e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:48.562 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.560+0000 7f40674e4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.560+0000 7f40674e4540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:11:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.560+0000 7f40674e4540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:11:48.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.560+0000 7f40674e4540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:11:48.652 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:48.648+0000 7f40674e4540 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-18T11:11:56.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-18T11:11:56.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:11:56.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T11:11:56.113 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd6 key to auth repository 2024-09-18T11:11:56.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:11:56.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T11:11:56.695 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.6 2024-09-18T11:11:56.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:11:56.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:11:56.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:11:56.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:11:56.722 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:56.716+0000 7fc0af9f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:56.745 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:56.740+0000 7fc0af9f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:56.746 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:56.744+0000 7fc0af9f5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:11:57.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T11:11:57.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:11:57.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T11:11:57.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:11:57.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:11:57.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:57.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:11:57.095 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:11:57.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:57.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:11:57.293 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:11:57.288+0000 7fc0af9f5540 -1 Falling back to public interface 2024-09-18T11:11:57.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:11:58.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:11:58.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:11:58.574 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:11:58.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:11:58.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:11:58.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:11:59.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:00.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:00.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:00.003 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:12:00.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:12:00.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:00.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:00.213 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:00.208+0000 7fc0af9f5540 -1 osd.6 0 log_to_monitors true 2024-09-18T11:12:00.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:01.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:01.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:01.437 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:12:01.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:12:01.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:01.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:01.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:02.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:02.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:02.928 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:12:02.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:12:02.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:02.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:03.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:04.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:04.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:04.515 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:12:04.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:12:04.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:04.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:05.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:06.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:06.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:06.106 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:12:06.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:12:06.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:06.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:06.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:07.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:07.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:07.540 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:12:07.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:12:07.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:07.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:08.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:09.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:09.112 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:12:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:12:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:09.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:10.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:10.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:10.555 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:12:10.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:12:10.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:10.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:11.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:12.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:12.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:12.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:12:12.222 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:12:12.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:12.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:12.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:13.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:13.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:13.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:12:13.783 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:12:13.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:13.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:14.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:15.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:15.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:15.208 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:12:15.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:12:15.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:15.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:15.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:16.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:16.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:16.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:12:16.789 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:12:16.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:16.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:17.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:18.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:18.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:18.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:12:18.292 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:12:18.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:18.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:18.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:19.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:19.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:19.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:12:19.772 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:12:19.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:19.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:20.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:21.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:21.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:21.647 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:12:21.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:12:21.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:21.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:22.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:23.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:23.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:23.663 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:12:23.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:12:23.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:23.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:24.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:12:25.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:12:25.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:25.641 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:12:25.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:12:25.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:25.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:12:26.238 INFO:tasks.workunit.client.0.smithi174.stdout:osd.6 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/163758,v1:127.0.0.1:6851/163758] [v2:127.0.0.1:6852/163758,v1:127.0.0.1:6853/163758] exists,up 2a855820-3e54-42f9-ac31-60b227e41e8b 2024-09-18T11:12:26.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:12:26.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:12:26.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:12:26.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:12:26.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:12:26.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:12:26.241 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:12:26.241 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:26.241 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:26.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:12:26.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:12:26.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:12:26.331 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:12:26.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:12:26.351 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:08:15.243+0000 7fd0b9749540 0 load: jerasure load: lrc 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-18T11:12:26.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:12:26.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-18T11:12:27.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:12:27.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:12:29.031 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:12:29.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:12:30.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:12:30.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:12:30.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:12:30.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:12:30.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:12:30.043 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:12:30.043 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:12:30.043 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:12:30.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:12:30.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:12:30.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:12:30.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:12:30.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:12:30.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:12:30.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:12:30.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:12:30.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:12:30.705 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:12:30.705 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:12:30.705 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:12:30.705 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:12:30.705 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:12:30.706 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:12:30.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:12:30.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:30.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:12:30.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803834 2024-09-18T11:12:30.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803834 2024-09-18T11:12:30.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834' 2024-09-18T11:12:30.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:30.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:12:31.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574895 2024-09-18T11:12:31.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574895 2024-09-18T11:12:31.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834 1-55834574895' 2024-09-18T11:12:31.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:31.011 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:12:31.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378667 2024-09-18T11:12:31.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378667 2024-09-18T11:12:31.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834 1-55834574895 2-81604378667' 2024-09-18T11:12:31.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:31.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:12:31.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084323 2024-09-18T11:12:31.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084323 2024-09-18T11:12:31.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834 1-55834574895 2-81604378667 3-120259084323' 2024-09-18T11:12:31.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:31.310 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:12:31.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724574 2024-09-18T11:12:31.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724574 2024-09-18T11:12:31.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834 1-55834574895 2-81604378667 3-120259084323 4-167503724574' 2024-09-18T11:12:31.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:31.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:12:31.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397524 2024-09-18T11:12:31.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397524 2024-09-18T11:12:31.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834 1-55834574895 2-81604378667 3-120259084323 4-167503724574 5-210453397524' 2024-09-18T11:12:31.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:31.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:12:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070476 2024-09-18T11:12:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070476 2024-09-18T11:12:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803834 1-55834574895 2-81604378667 3-120259084323 4-167503724574 5-210453397524 6-253403070476' 2024-09-18T11:12:31.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:31.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803834 2024-09-18T11:12:31.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:31.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:12:31.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803834 2024-09-18T11:12:31.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:31.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803834 2024-09-18T11:12:31.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-09-18T11:12:31.763 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803834 2024-09-18T11:12:31.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:12:32.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803834 -lt 25769803834 2024-09-18T11:12:32.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:32.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574895 2024-09-18T11:12:32.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:32.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:12:32.184 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574895 2024-09-18T11:12:32.184 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:32.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574895 2024-09-18T11:12:32.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-18T11:12:32.186 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574895 2024-09-18T11:12:32.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:12:32.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574895 -lt 55834574895 2024-09-18T11:12:32.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:32.719 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378667 2024-09-18T11:12:32.719 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:32.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:12:32.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378667 2024-09-18T11:12:32.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:32.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378667 2024-09-18T11:12:32.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378667' 2024-09-18T11:12:32.725 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378667 2024-09-18T11:12:32.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:12:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378667 -lt 81604378667 2024-09-18T11:12:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:33.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084323 2024-09-18T11:12:33.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:33.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:12:33.133 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084323 2024-09-18T11:12:33.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:33.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084323 2024-09-18T11:12:33.136 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084323 2024-09-18T11:12:33.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084323' 2024-09-18T11:12:33.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:12:33.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084323 -lt 120259084323 2024-09-18T11:12:33.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:33.555 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724574 2024-09-18T11:12:33.555 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:33.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:12:33.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:33.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724574 2024-09-18T11:12:33.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724574 2024-09-18T11:12:33.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724574' 2024-09-18T11:12:33.559 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724574 2024-09-18T11:12:33.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:12:33.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724574 -lt 167503724574 2024-09-18T11:12:33.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:33.966 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-210453397524 2024-09-18T11:12:33.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:33.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:12:33.970 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-210453397524 2024-09-18T11:12:33.970 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:33.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397524 2024-09-18T11:12:33.971 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 210453397524 2024-09-18T11:12:33.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 210453397524' 2024-09-18T11:12:33.972 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:12:34.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397525 -lt 210453397524 2024-09-18T11:12:34.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:34.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-253403070476 2024-09-18T11:12:34.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:34.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:12:34.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-253403070476 2024-09-18T11:12:34.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:34.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070476 2024-09-18T11:12:34.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 253403070476' 2024-09-18T11:12:34.354 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 253403070476 2024-09-18T11:12:34.354 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:12:34.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070477 -lt 253403070476 2024-09-18T11:12:34.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:12:34.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:12:34.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:12:35.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:12:35.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:12:35.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:12:35.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:12:35.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:12:35.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:12:35.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:12:35.419 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:12:35.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:12:35.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:12:35.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:12:35.861 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:12:36.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:12:36.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:12:36.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:12:36.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:12:36.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:12:36.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:12:36.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-09-18T11:12:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-09-18T11:12:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:12:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:12:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-09-18T11:12:36.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:12:36.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-09-18T11:12:37.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:12:37.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:12:37.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-18T11:12:37.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:12:37.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:12:37.243 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:12:37.244 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:12:37.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:12:37.640 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:12:37.640 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:12:37.640 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:12:37.640 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:12:37.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-18T11:12:37.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:12:37.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T11:12:37.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:12:37.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T11:12:37.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:12:37.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T11:12:37.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:12:37.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T11:12:37.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:12:37.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:12:37.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:12:37.645 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T11:12:37.645 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:12:37.645 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:12:37.646 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:12:37.646 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:37.646 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:37.646 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:12:37.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:12:37.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:12:37.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:12:37.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:12:37.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T11:12:37.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:12:37.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:12:37.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:12:37.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:37.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:37.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:12:37.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:12:37.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-18T11:12:37.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-09-18T11:12:38.474 INFO:tasks.workunit.client.0.smithi174.stderr:osd.0 is already out. 2024-09-18T11:12:38.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-09-18T11:12:38.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:12:38.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:12:38.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:12:38.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:12:38.490 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:12:38.490 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:12:38.490 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:12:38.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:12:38.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:12:38.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:12:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:12:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:12:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:12:38.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:12:38.699 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:12:39.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:12:39.282 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:39.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:12:39.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803838 2024-09-18T11:12:39.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803838 2024-09-18T11:12:39.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838' 2024-09-18T11:12:39.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:39.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:12:39.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574899 2024-09-18T11:12:39.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574899 2024-09-18T11:12:39.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838 1-55834574899' 2024-09-18T11:12:39.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:39.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:12:39.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378671 2024-09-18T11:12:39.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378671 2024-09-18T11:12:39.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838 1-55834574899 2-81604378671' 2024-09-18T11:12:39.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:39.856 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:12:40.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084327 2024-09-18T11:12:40.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084327 2024-09-18T11:12:40.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838 1-55834574899 2-81604378671 3-120259084327' 2024-09-18T11:12:40.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:40.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:12:40.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724578 2024-09-18T11:12:40.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724578 2024-09-18T11:12:40.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838 1-55834574899 2-81604378671 3-120259084327 4-167503724578' 2024-09-18T11:12:40.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:40.290 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:12:40.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397528 2024-09-18T11:12:40.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397528 2024-09-18T11:12:40.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838 1-55834574899 2-81604378671 3-120259084327 4-167503724578 5-210453397528' 2024-09-18T11:12:40.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:12:40.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:12:40.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=253403070480 2024-09-18T11:12:40.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 253403070480 2024-09-18T11:12:40.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803838 1-55834574899 2-81604378671 3-120259084327 4-167503724578 5-210453397528 6-253403070480' 2024-09-18T11:12:40.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:40.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803838 2024-09-18T11:12:40.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:40.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:12:40.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803838 2024-09-18T11:12:40.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:40.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803838 2024-09-18T11:12:40.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-09-18T11:12:40.591 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803838 2024-09-18T11:12:40.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:12:41.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803838 -lt 25769803838 2024-09-18T11:12:41.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:41.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574899 2024-09-18T11:12:41.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:41.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:12:41.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574899 2024-09-18T11:12:41.061 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:41.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574899 2024-09-18T11:12:41.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574899' 2024-09-18T11:12:41.064 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574899 2024-09-18T11:12:41.065 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:12:41.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574897 -lt 55834574899 2024-09-18T11:12:41.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:12:42.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:12:42.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:12:42.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574899 -lt 55834574899 2024-09-18T11:12:42.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:42.899 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:42.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378671 2024-09-18T11:12:42.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:12:42.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378671 2024-09-18T11:12:42.902 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:42.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378671 2024-09-18T11:12:42.903 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378671 2024-09-18T11:12:42.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378671' 2024-09-18T11:12:42.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:12:43.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378671 -lt 81604378671 2024-09-18T11:12:43.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:43.331 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084327 2024-09-18T11:12:43.331 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:43.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:12:43.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:43.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084327 2024-09-18T11:12:43.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084327 2024-09-18T11:12:43.336 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084327 2024-09-18T11:12:43.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084327' 2024-09-18T11:12:43.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:12:43.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084328 -lt 120259084327 2024-09-18T11:12:43.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:43.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724578 2024-09-18T11:12:43.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:43.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:12:43.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724578 2024-09-18T11:12:43.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:43.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724578 2024-09-18T11:12:43.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724578' 2024-09-18T11:12:43.790 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724578 2024-09-18T11:12:43.790 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:12:44.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724578 -lt 167503724578 2024-09-18T11:12:44.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:44.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-210453397528 2024-09-18T11:12:44.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:44.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:12:44.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-210453397528 2024-09-18T11:12:44.189 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:44.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397528 2024-09-18T11:12:44.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 210453397528' 2024-09-18T11:12:44.191 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 210453397528 2024-09-18T11:12:44.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:12:44.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397529 -lt 210453397528 2024-09-18T11:12:44.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:12:44.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-253403070480 2024-09-18T11:12:44.663 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:12:44.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:12:44.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-253403070480 2024-09-18T11:12:44.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:12:44.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=253403070480 2024-09-18T11:12:44.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 253403070480' 2024-09-18T11:12:44.670 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 253403070480 2024-09-18T11:12:44.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:12:45.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 253403070481 -lt 253403070480 2024-09-18T11:12:45.055 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:12:45.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:12:45.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:12:45.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:12:45.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:12:45.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:12:45.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:12:45.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:12:45.567 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:12:45.567 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:12:45.567 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:12:45.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:12:45.951 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:12:45.951 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:12:45.951 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:12:46.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:12:46.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:12:46.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:12:46.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-09-18T11:12:46.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:12:46.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:12:46.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-09-18T11:12:46.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:12:46.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:12:46.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-09-18T11:12:46.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:12:46.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:12:46.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:12:46.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:12:46.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:12:47.271 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:12:47.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:12:48.262 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:12:48.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:12:48.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:12:48.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:12:48.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:12:48.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:12:48.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:12:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:12:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:12:48.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:12:48.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:12:48.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:12:48.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:12:48.400 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:12:48.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:12:48.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:12:48.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:12:48.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:12:48.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:12:48.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:12:48.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:12:48.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:12:48.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:12:48.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:12:48.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:12:48.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:48.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:48.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:12:48.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:12:48.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:12:48.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:12:48.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:12:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:12:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:12:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:12:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:12:48.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:12:48.450 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:12:48.450 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:12:48.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:12:48.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:12:48.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:12:48.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:12:48.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:12:48.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:12:48.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:12:48.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:12:48.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:12:48.458 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:12:48.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:12:48.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:12:48.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:12:48.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:12:48.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:12:48.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:12:48.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:12:48.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:12:48.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:48.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:48.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:12:48.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:12:48.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:12:48.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:12:48.469 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:12:48.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:48.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:12:48.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:12:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:12:48.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:12:48.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:12:48.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:12:48.842 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:12:48.842 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:48.842 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:48.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:12:48.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:12:49.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:12:49.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:12:49.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:12:49.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:12:49.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:12:49.125 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:12:49.126 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:12:49.126 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:12:49.126 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:12:49.127 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:12:49.127 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:49.127 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:49.127 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:12:49.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:12:49.128 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:12:49.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:12:49.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:12:49.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:12:49.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:12:49.219 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:12:49.219 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:12:49.220 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:12:49.220 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:12:49.220 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:12:49.220 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:49.221 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:49.221 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:12:49.222 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:12:49.222 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:12:49.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:12:49.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:12:49.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:12:49.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:12:49.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:12:49.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:12:49.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:12:50.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:12:50.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:12:50.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:12:50.206 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:12:50.206 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:50.206 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:50.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:12:50.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:12:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:12:50.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:12:50.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:12:50.609 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:12:50.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:12:51.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:12:51.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:12:51.622 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:12:51.623 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:12:51.623 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:51.623 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:51.623 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:12:51.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:12:51.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:12:51.703 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:12:51.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:12:51.713 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:12:48.879+0000 7fb6802d0a00 0 load: jerasure load: lrc 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-09-18T11:12:51.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-09-18T11:12:51.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:12:51.716 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-09-18T11:12:51.717 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-09-18T11:12:51.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:12:51.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:12:51.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:12:51.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:12:51.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:12:51.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:12:51.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59eea00c-515f-46a5-adb5-a71167b211c4 2024-09-18T11:12:51.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 59eea00c-515f-46a5-adb5-a71167b211c4' 2024-09-18T11:12:51.726 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 59eea00c-515f-46a5-adb5-a71167b211c4 2024-09-18T11:12:51.726 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:12:51.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCztepm6xmMLBAA99+hvrMnFme8heQ8/RjRww== 2024-09-18T11:12:51.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCztepm6xmMLBAA99+hvrMnFme8heQ8/RjRww=="}' 2024-09-18T11:12:51.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59eea00c-515f-46a5-adb5-a71167b211c4 -i td/test-erasure-eio/0/new.json 2024-09-18T11:12:52.117 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:12:52.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:12:52.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCztepm6xmMLBAA99+hvrMnFme8heQ8/RjRww== --osd-uuid 59eea00c-515f-46a5-adb5-a71167b211c4 2024-09-18T11:12:52.168 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.163+0000 7ff07541c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:12:52.170 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.167+0000 7ff07541c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:12:52.172 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.167+0000 7ff07541c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:12:52.172 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.167+0000 7ff07541c540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:12:52.173 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.167+0000 7ff07541c540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:12:52.173 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.167+0000 7ff07541c540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:12:52.244 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:52.239+0000 7ff07541c540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:12:58.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:12:58.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:12:58.325 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:12:58.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:12:58.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:12:58.875 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:12:58.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:12:58.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:12:58.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:12:58.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:12:58.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:12:58.913 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:58.907+0000 7f2ae820a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:12:58.916 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:58.911+0000 7f2ae820a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:12:58.917 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:58.911+0000 7f2ae820a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:12:59.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:12:59.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:12:59.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:12:59.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:12:59.505 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:12:59.499+0000 7f2ae820a540 -1 Falling back to public interface 2024-09-18T11:12:59.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:00.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:00.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:00.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:13:00.686 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:13:00.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:00.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:01.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:02.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:02.151 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:13:02.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:13:02.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:02.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:02.198 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:02.195+0000 7f2ae820a540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:13:02.623 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:03.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:03.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:03.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:13:03.625 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:13:03.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:03.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:04.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:05.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:05.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:05.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:13:05.119 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:13:05.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:05.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:05.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:06.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:06.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:06.606 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:13:06.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:13:06.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:06.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:07.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:08.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:08.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:08.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:13:08.038 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:13:08.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:08.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:08.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:09.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:09.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:09.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:13:09.484 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:13:09.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:09.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:09.978 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:10.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:10.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:10.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:13:10.980 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:13:10.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:10.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:11.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:12.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:12.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:12.452 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:13:12.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:13:12.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:12.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:12.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:13.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:13.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:13.926 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:13:13.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:13:13.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:13.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:14.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:15.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:15.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:15.384 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:13:15.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:13:15.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:15.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:16.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:17.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:17.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:17.074 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:13:17.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:13:17.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:17.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:18.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:19.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:19.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:19.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:13:19.111 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:13:19.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:19.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:19.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:20.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:20.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:20.809 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:13:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:13:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:20.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:21.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:22.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:22.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:22.932 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:13:22.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:13:22.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:22.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:23.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:24.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:24.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:24.632 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:13:24.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:13:24.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:24.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:25.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:26.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:26.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:26.185 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:13:26.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:13:26.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:26.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:13:26.731 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/167405,v1:127.0.0.1:6803/167405] [v2:127.0.0.1:6804/167405,v1:127.0.0.1:6805/167405] exists,up 59eea00c-515f-46a5-adb5-a71167b211c4 2024-09-18T11:13:26.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:13:26.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:13:26.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:13:26.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:13:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:13:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:13:26.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:13:26.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:13:26.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:13:26.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:13:26.736 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:13:26.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:13:26.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:13:26.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:13:26.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:13:26.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:13:26.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=deefa598-3327-48ef-83da-89e9ee214ba5 2024-09-18T11:13:26.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 deefa598-3327-48ef-83da-89e9ee214ba5' 2024-09-18T11:13:26.740 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 deefa598-3327-48ef-83da-89e9ee214ba5 2024-09-18T11:13:26.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:13:26.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWtepm5C57LRAApMTOL93Cv+awaFwiAY8xrg== 2024-09-18T11:13:26.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWtepm5C57LRAApMTOL93Cv+awaFwiAY8xrg=="}' 2024-09-18T11:13:26.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new deefa598-3327-48ef-83da-89e9ee214ba5 -i td/test-erasure-eio/1/new.json 2024-09-18T11:13:27.349 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:13:27.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:13:27.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWtepm5C57LRAApMTOL93Cv+awaFwiAY8xrg== --osd-uuid deefa598-3327-48ef-83da-89e9ee214ba5 2024-09-18T11:13:27.407 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.403+0000 7ff6cfca0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:13:27.430 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.423+0000 7ff6cfca0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:13:27.432 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.427+0000 7ff6cfca0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:13:27.433 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.427+0000 7ff6cfca0540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:13:27.433 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.427+0000 7ff6cfca0540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:13:27.433 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.427+0000 7ff6cfca0540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:13:27.529 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:27.527+0000 7ff6cfca0540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:13:33.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:13:33.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:13:33.789 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:13:33.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:13:33.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:13:34.337 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:13:34.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:13:34.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:13:34.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:13:34.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:13:34.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:13:34.376 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:34.371+0000 7f3e1593b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:13:34.427 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:34.423+0000 7f3e1593b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:13:34.428 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:34.423+0000 7f3e1593b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:13:34.725 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:34.719+0000 7f3e1593b540 -1 Falling back to public interface 2024-09-18T11:13:34.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:13:34.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:13:34.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:13:34.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:13:34.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:13:34.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:34.838 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:13:34.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:13:34.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:34.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:35.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:36.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:36.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:36.248 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:13:36.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:13:36.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:36.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:36.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:37.636 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:13:37.631+0000 7f3e1593b540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:13:37.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:37.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:37.703 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:13:37.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:13:37.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:37.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:38.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:39.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:39.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:39.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:13:39.394 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:13:39.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:39.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:39.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:40.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:40.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:40.885 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:13:40.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:13:40.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:40.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:41.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:42.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:42.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:42.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:13:42.487 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:13:42.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:42.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:42.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:43.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:43.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:43.918 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:13:43.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:13:43.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:43.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:44.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:45.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:45.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:45.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:13:45.519 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:13:45.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:45.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:45.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:46.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:46.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:47.000 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:13:47.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:13:47.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:47.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:47.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:48.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:48.474 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:13:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:13:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:48.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:49.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:50.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:50.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:50.063 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:13:50.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:13:50.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:50.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:50.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:51.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:51.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:51.531 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:13:51.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:13:51.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:51.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:52.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:53.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:53.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:53.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:13:53.211 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:13:53.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:53.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:54.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:55.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:55.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:55.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:13:55.187 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:13:55.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:55.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:55.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:56.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:56.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:56.968 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:13:56.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:13:56.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:56.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:57.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:58.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:58.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:58.408 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:13:58.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:13:58.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:58.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:13:58.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:13:59.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:13:59.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:13:59.896 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:13:59.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:13:59.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:13:59.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:14:00.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:01.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:01.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:01.441 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:14:01.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:14:01.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:01.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:14:01.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:02.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:02.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:02.972 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:14:02.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:14:02.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:02.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:14:03.456 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/169182,v1:127.0.0.1:6811/169182] [v2:127.0.0.1:6812/169182,v1:127.0.0.1:6813/169182] exists,up deefa598-3327-48ef-83da-89e9ee214ba5 2024-09-18T11:14:03.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:14:03.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:14:03.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:14:03.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:14:03.460 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:14:03.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:14:03.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:14:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5a24a5a-8e39-49b1-b462-b3fe026fc12c 2024-09-18T11:14:03.463 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 d5a24a5a-8e39-49b1-b462-b3fe026fc12c 2024-09-18T11:14:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d5a24a5a-8e39-49b1-b462-b3fe026fc12c' 2024-09-18T11:14:03.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:14:03.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7tepmdk7IHBAAdRffw6+FQL2mz7pzmpDBSA== 2024-09-18T11:14:03.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7tepmdk7IHBAAdRffw6+FQL2mz7pzmpDBSA=="}' 2024-09-18T11:14:03.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5a24a5a-8e39-49b1-b462-b3fe026fc12c -i td/test-erasure-eio/2/new.json 2024-09-18T11:14:04.075 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:14:04.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:14:04.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7tepmdk7IHBAAdRffw6+FQL2mz7pzmpDBSA== --osd-uuid d5a24a5a-8e39-49b1-b462-b3fe026fc12c 2024-09-18T11:14:04.125 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.118+0000 7f64bc29a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:04.130 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.122+0000 7f64bc29a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:04.132 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.126+0000 7f64bc29a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:04.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.126+0000 7f64bc29a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:14:04.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.126+0000 7f64bc29a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:14:04.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.126+0000 7f64bc29a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:14:04.230 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:04.226+0000 7f64bc29a540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:14:10.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:14:10.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:14:10.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:14:10.802 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:14:10.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:14:11.341 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:14:11.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:14:11.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:14:11.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:14:11.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:14:11.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:14:11.380 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:11.374+0000 7fb498d91540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:11.404 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:11.398+0000 7fb498d91540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:11.405 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:11.398+0000 7fb498d91540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:11.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:14:11.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:14:11.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:14:11.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:14:11.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:14:11.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:11.854 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:14:11.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:14:11.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:11.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:11.960 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:11.954+0000 7fb498d91540 -1 Falling back to public interface 2024-09-18T11:14:12.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:13.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:13.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:13.302 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:14:13.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:14:13.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:13.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:13.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:14.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:14.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:14.767 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:14:14.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:14:14.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:14.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:14.796 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:14.790+0000 7fb498d91540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:14:15.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:16.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:16.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:16.284 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:14:16.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:14:16.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:16.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:16.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:17.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:17.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:17.751 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:14:17.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:14:17.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:17.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:18.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:19.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:19.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:19.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:14:19.186 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:14:19.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:19.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:19.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:20.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:20.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:20.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:14:20.792 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:14:20.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:20.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:21.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:22.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:22.219 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:14:22.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:14:22.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:22.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:22.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:23.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:23.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:23.718 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:14:23.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:14:23.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:23.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:24.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:25.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:25.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:25.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:14:25.134 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:14:25.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:25.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:25.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:26.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:26.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:26.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:14:26.673 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:14:26.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:26.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:27.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:28.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:28.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:28.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:14:28.093 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:14:28.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:28.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:28.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:29.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:29.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:29.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:14:29.852 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:14:29.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:29.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:30.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:31.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:31.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:31.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:14:31.602 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:14:31.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:31.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:32.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:33.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:33.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:33.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:14:33.342 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:14:33.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:33.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:34.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:35.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:35.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:35.398 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:14:35.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:14:35.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:35.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:36.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:37.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:37.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:37.137 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:14:37.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:14:37.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:37.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:37.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:38.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:38.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:38.755 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:14:38.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:14:38.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:38.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:14:39.182 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/170982,v1:127.0.0.1:6819/170982] [v2:127.0.0.1:6820/170982,v1:127.0.0.1:6821/170982] exists,up d5a24a5a-8e39-49b1-b462-b3fe026fc12c 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:14:39.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:14:39.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:14:39.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:14:39.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:14:39.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:14:39.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:14:39.188 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:14:39.188 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:14:39.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:14:39.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:14:39.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:14:39.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcdad1b5-9357-47f8-b05d-23e50238c86b 2024-09-18T11:14:39.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 fcdad1b5-9357-47f8-b05d-23e50238c86b' 2024-09-18T11:14:39.190 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 fcdad1b5-9357-47f8-b05d-23e50238c86b 2024-09-18T11:14:39.190 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:14:39.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAftupmhOCnDBAApF7uXMiULYlBnfGhI05qbw== 2024-09-18T11:14:39.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAftupmhOCnDBAApF7uXMiULYlBnfGhI05qbw=="}' 2024-09-18T11:14:39.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcdad1b5-9357-47f8-b05d-23e50238c86b -i td/test-erasure-eio/3/new.json 2024-09-18T11:14:39.899 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:14:39.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:14:39.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAftupmhOCnDBAApF7uXMiULYlBnfGhI05qbw== --osd-uuid fcdad1b5-9357-47f8-b05d-23e50238c86b 2024-09-18T11:14:39.955 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:39.950+0000 7f5675c42540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:39.966 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:39.962+0000 7f5675c42540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:39.967 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:39.962+0000 7f5675c42540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:39.968 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:39.962+0000 7f5675c42540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:14:39.968 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:39.962+0000 7f5675c42540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:14:39.968 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:39.962+0000 7f5675c42540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:14:40.108 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:40.102+0000 7f5675c42540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:14:47.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:14:47.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:14:47.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:14:47.109 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:14:47.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:14:47.679 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:14:47.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:14:47.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:14:47.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:14:47.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:14:47.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:14:47.717 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:47.710+0000 7fcfb1d6f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:47.720 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:47.714+0000 7fcfb1d6f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:47.722 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:47.714+0000 7fcfb1d6f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:14:48.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:14:48.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:14:48.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:14:48.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:14:48.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:14:48.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:48.091 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:14:48.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:14:48.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:48.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:48.285 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:48.278+0000 7fcfb1d6f540 -1 Falling back to public interface 2024-09-18T11:14:48.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:49.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:49.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:49.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:14:49.525 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:14:49.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:49.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:49.930 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:50.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:50.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:50.933 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:14:50.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:14:50.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:50.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:51.106 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:14:51.098+0000 7fcfb1d6f540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:14:51.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:52.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:52.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:52.448 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:14:52.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:14:52.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:52.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:53.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:54.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:54.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:54.372 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:14:54.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:14:54.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:54.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:54.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:55.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:55.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:55.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:14:55.983 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:14:55.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:55.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:56.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:57.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:57.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:57.411 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:14:57.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:14:57.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:57.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:58.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:14:59.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:14:59.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:14:59.027 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:14:59.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:14:59.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:14:59.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:14:59.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:00.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:00.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:00.490 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:15:00.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:15:00.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:00.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:00.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:01.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:01.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:01.911 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:15:01.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:15:01.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:01.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:02.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:03.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:03.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:03.350 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:15:03.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:15:03.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:03.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:03.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:04.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:04.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:04.991 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:15:04.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:15:04.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:04.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:05.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:06.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:06.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:06.533 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:15:06.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:15:06.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:06.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:07.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:08.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:08.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:08.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:15:08.166 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:15:08.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:08.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:08.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:15:09.764 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:15:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:10.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:11.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:11.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:11.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:15:11.223 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:15:11.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:11.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:12.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:13.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:13.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:13.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:15:13.108 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:15:13.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:13.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:14.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:15.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:15.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:15.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:15:15.095 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:15:15.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:15.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:17.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:17.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:17.047 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:15:17.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:15:17.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:17.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:15:17.496 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/172756,v1:127.0.0.1:6827/172756] [v2:127.0.0.1:6828/172756,v1:127.0.0.1:6829/172756] exists,up fcdad1b5-9357-47f8-b05d-23e50238c86b 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:15:17.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:15:17.500 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:15:17.501 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:15:17.501 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:15:17.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:15:17.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:15:17.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:15:17.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T11:15:17.503 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:15:17.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ac90667-9fcf-42d9-aac0-6ab8d53cd13b 2024-09-18T11:15:17.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0ac90667-9fcf-42d9-aac0-6ab8d53cd13b' 2024-09-18T11:15:17.504 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 0ac90667-9fcf-42d9-aac0-6ab8d53cd13b 2024-09-18T11:15:17.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:15:17.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFtupmWKJ5HxAA9fPFARLrFMSB5uO018nXkw== 2024-09-18T11:15:17.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFtupmWKJ5HxAA9fPFARLrFMSB5uO018nXkw=="}' 2024-09-18T11:15:17.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ac90667-9fcf-42d9-aac0-6ab8d53cd13b -i td/test-erasure-eio/4/new.json 2024-09-18T11:15:18.364 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:15:18.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T11:15:18.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFtupmWKJ5HxAA9fPFARLrFMSB5uO018nXkw== --osd-uuid 0ac90667-9fcf-42d9-aac0-6ab8d53cd13b 2024-09-18T11:15:18.421 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.413+0000 7fc6f9168540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:15:18.519 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.513+0000 7fc6f9168540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:15:18.521 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.513+0000 7fc6f9168540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:15:18.522 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.513+0000 7fc6f9168540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:15:18.522 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.513+0000 7fc6f9168540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:15:18.522 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.513+0000 7fc6f9168540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:15:18.639 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:18.633+0000 7fc6f9168540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T11:15:25.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T11:15:25.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:15:25.301 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T11:15:25.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T11:15:25.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:15:25.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T11:15:25.856 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T11:15:25.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:15:25.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:15:25.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:15:25.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:15:25.888 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:25.881+0000 7efd2a7ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:15:25.890 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:25.881+0000 7efd2a7ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:15:25.892 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:25.885+0000 7efd2a7ef540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:15:26.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T11:15:26.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:15:26.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T11:15:26.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:15:26.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:15:26.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:26.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:15:26.301 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:15:26.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:26.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:26.445 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:26.437+0000 7efd2a7ef540 -1 Falling back to public interface 2024-09-18T11:15:26.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:27.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:27.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:27.721 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:15:27.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:15:27.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:27.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:28.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:29.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:29.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:29.136 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:15:29.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:15:29.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:29.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:29.197 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:15:29.189+0000 7efd2a7ef540 -1 osd.4 0 log_to_monitors true 2024-09-18T11:15:29.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:30.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:30.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:30.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:15:30.658 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:15:30.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:30.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:31.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:32.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:32.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:32.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:15:32.298 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:15:32.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:32.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:32.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:33.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:33.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:33.867 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:15:33.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:15:33.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:33.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:34.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:35.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:35.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:35.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:15:35.436 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:15:35.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:35.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:35.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:36.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:36.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:36.992 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:15:36.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:15:36.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:36.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:37.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:38.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:38.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:38.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:15:38.444 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:15:38.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:38.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:39.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:40.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:40.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:40.038 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:15:40.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:15:40.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:40.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:40.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:41.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:41.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:41.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:15:41.660 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:15:41.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:41.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:42.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:15:43.151 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:15:43.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:43.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:44.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:45.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:45.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:45.106 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:15:45.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:15:45.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:45.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:45.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:46.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:46.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:46.666 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:15:46.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:15:46.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:46.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:47.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:48.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:48.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:48.754 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:15:48.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:15:48.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:48.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:49.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:50.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:50.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:50.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:15:50.288 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:15:50.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:50.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:51.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:52.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:52.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:52.259 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:15:52.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:15:52.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:52.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:52.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:15:53.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:15:53.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:15:53.864 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:15:53.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:15:53.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:15:53.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:15:54.659 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/174560,v1:127.0.0.1:6835/174560] [v2:127.0.0.1:6836/174560,v1:127.0.0.1:6837/174560] exists,up 0ac90667-9fcf-42d9-aac0-6ab8d53cd13b 2024-09-18T11:15:54.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:15:54.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:15:54.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:15:54.660 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:15:54.661 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:15:54.661 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:15:54.661 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:15:54.662 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:15:54.662 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:15:54.662 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:15:54.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:15:54.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:15:54.745 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:15:54.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:15:54.761 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:12:59.511+0000 7f2ae820a540 0 load: jerasure load: lrc 2024-09-18T11:15:54.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:15:54.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-18T11:15:55.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:15:55.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:15:57.402 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:15:57.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:15:58.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:15:58.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:15:58.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:15:58.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:15:58.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:15:58.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:15:58.414 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:15:58.414 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:15:58.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:15:58.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:15:58.415 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:15:58.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:15:58.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:15:58.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:15:58.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:15:58.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:15:59.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr:4' 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:15:59.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:15:59.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803817 2024-09-18T11:15:59.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803817 2024-09-18T11:15:59.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817' 2024-09-18T11:15:59.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:15:59.374 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:15:59.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542176 2024-09-18T11:15:59.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542176 2024-09-18T11:15:59.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-60129542176' 2024-09-18T11:15:59.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:15:59.748 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:15:59.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345945 2024-09-18T11:15:59.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345945 2024-09-18T11:15:59.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-60129542176 2-85899345945' 2024-09-18T11:15:59.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:15:59.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:16:00.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051605 2024-09-18T11:16:00.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051605 2024-09-18T11:16:00.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-60129542176 2-85899345945 3-124554051605' 2024-09-18T11:16:00.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:16:00.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:16:00.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724561 2024-09-18T11:16:00.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724561 2024-09-18T11:16:00.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-60129542176 2-85899345945 3-124554051605 4-167503724561' 2024-09-18T11:16:00.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:16:00.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803817 2024-09-18T11:16:00.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:16:00.207 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:16:00.209 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803817 2024-09-18T11:16:00.209 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:16:00.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803817 2024-09-18T11:16:00.211 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803817 2024-09-18T11:16:00.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-09-18T11:16:00.211 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:16:00.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803817 -lt 25769803817 2024-09-18T11:16:00.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:16:00.615 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542176 2024-09-18T11:16:00.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:16:00.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:16:00.619 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542176 2024-09-18T11:16:00.619 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:16:00.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542176 2024-09-18T11:16:00.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542176' 2024-09-18T11:16:00.621 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 60129542176 2024-09-18T11:16:00.621 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:16:01.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542174 -lt 60129542176 2024-09-18T11:16:01.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:16:02.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:16:02.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:16:02.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542176 -lt 60129542176 2024-09-18T11:16:02.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:16:02.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345945 2024-09-18T11:16:02.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:16:02.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:16:02.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345945 2024-09-18T11:16:02.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:16:02.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345945 2024-09-18T11:16:02.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345945' 2024-09-18T11:16:02.445 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 85899345945 2024-09-18T11:16:02.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:16:02.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345945 -lt 85899345945 2024-09-18T11:16:02.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:16:02.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051605 2024-09-18T11:16:02.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:16:02.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:16:02.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051605 2024-09-18T11:16:02.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:16:02.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051605 2024-09-18T11:16:02.903 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051605 2024-09-18T11:16:02.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051605' 2024-09-18T11:16:02.904 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:16:03.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051605 -lt 124554051605 2024-09-18T11:16:03.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:16:03.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724561 2024-09-18T11:16:03.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:16:03.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:16:03.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:16:03.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724561 2024-09-18T11:16:03.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724561 2024-09-18T11:16:03.313 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724561 2024-09-18T11:16:03.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724561' 2024-09-18T11:16:03.313 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:16:03.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724561 -lt 167503724561 2024-09-18T11:16:03.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:16:03.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:16:03.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:16:04.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:16:04.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:16:04.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:16:04.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:16:04.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:16:04.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:16:04.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:16:04.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:16:04.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:16:04.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:16:04.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:16:04.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:16:05.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:16:05.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:16:05.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:16:05.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:59.000275+0000 0'0 46:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-18T11:15:56.962778+0000 0'0 2024-09-18T11:15:56.962778+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:46:55.402099+0000 0 0 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.712882+0000 0'0 46:15 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:49:54.725022+0000 0 0 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.277336+0000 0'0 45:106 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:40:14.216542+0000 0 0 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.570254+0000 0'0 46:15 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:07:02.049217+0000 0 0 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:14:40.575611+0000 0'0 46:42 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:08:01.469829+0000 0 0 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T11:16:05.593 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T11:16:05.596 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:16:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:16:05.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:05.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:16:05.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-09-18T11:16:05.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:16:05.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:16:05.790 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:16:05.790 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:16:06.192 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:16:06.192 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:16:06.192 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:16:06.193 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:16:06.193 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:16:06.193 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-09-18T11:16:06.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-09-18T11:16:06.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-09-18T11:16:06.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:16:06.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:16:06.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:16:06.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:16:06.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:16:06.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:16:07.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:16:07.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-09-18T11:16:08.094 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already down. 2024-09-18T11:16:08.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-09-18T11:16:08.838 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-18T11:16:08.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 1 0 1 0 0 4096 0 0 1 0 1 activating+undersized 2024-09-18T11:16:07.427045+0000 47'1 50:19 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-09-18T11:15:56.962778+0000 0'0 2024-09-18T11:15:56.962778+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:31:32.653512+0000 0 0 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.712882+0000 0'0 46:15 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T22:49:54.725022+0000 0 0 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.277336+0000 0'0 46:107 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:40:14.216542+0000 0 0 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.570254+0000 0'0 46:15 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:07:02.049217+0000 0 0 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 activating+undersized 2024-09-18T11:16:07.096808+0000 0'0 49:47 [1,0] 1 [1,0] 1 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T16:55:59.835967+0000 0 0 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T11:16:09.275 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T11:16:09.278 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T11:16:09.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-09-18T11:16:09.289 INFO:tasks.workunit.client.0.smithi174.stderr:4+0 records in 2024-09-18T11:16:09.289 INFO:tasks.workunit.client.0.smithi174.stderr:4+0 records out 2024-09-18T11:16:09.289 INFO:tasks.workunit.client.0.smithi174.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000290633 s, 14.1 MB/s 2024-09-18T11:16:09.290 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-09-18T11:16:09.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:09.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:09.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:09.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:09.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:09.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:10.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:10.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:10.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:10.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:10.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:10.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:10.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:11.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:11.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:11.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:11.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:11.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:11.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:12.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:12.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:12.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:12.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:12.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:12.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:12.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:12.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:13.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:13.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:13.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:13.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:13.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:13.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:13.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:13.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:14.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:14.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:14.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:14.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:14.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:15.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:15.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:15.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:15.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:15.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:15.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:15.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:15.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:16.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:16.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:16.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:16.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:16.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:16.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:16.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:16.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:17.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:17.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:17.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:17.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:17.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:17.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:17.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:17.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:18.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:18.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:18.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:18.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:18.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:18.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:18.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:18.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:19.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:19.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:19.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:19.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:19.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:19.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:20.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:20.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:20.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:20.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:20.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:20.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:20.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:20.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:21.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:21.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:21.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:21.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:21.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:21.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:21.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:21.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:22.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:22.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:22.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:22.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:22.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:22.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:22.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:22.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:23.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:23.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:23.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:23.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:23.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:23.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:24.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:24.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:24.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:24.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:24.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:24.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:24.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:24.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:25.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:25.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:25.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:25.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:25.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:25.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:25.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:25.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:26.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:26.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:26.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:26.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:26.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:26.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:26.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:26.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:27.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:27.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:27.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:27.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:27.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:27.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:27.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:27.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:28.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:28.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:28.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:28.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:28.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:28.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:29.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:29.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:29.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:29.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:29.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:29.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:29.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:29.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:30.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:30.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:30.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:30.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:30.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:30.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:30.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:30.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:31.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:31.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:31.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:31.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:31.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:31.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:32.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:32.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:32.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:32.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:32.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:32.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:32.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:32.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:33.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:33.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:33.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:33.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:33.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:33.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:33.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:33.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:34.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:34.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:34.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:34.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:34.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:34.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:34.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:34.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:35.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:35.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:35.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:16:35.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-09-18T11:16:35.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-09-18T11:16:35.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:16:35.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:16:35.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:16:35.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj75 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:16:35.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-18T11:16:35.667 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-18T11:16:35.667 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:16:36.087 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:16:36.087 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:16:36.087 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:16:36.087 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:16:36.087 INFO:tasks.workunit.client.0.smithi174.stderr:2147483647' 2024-09-18T11:16:36.087 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-18T11:16:36.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:16:36.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T11:16:36.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:16:36.088 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T11:16:36.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:16:36.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T11:16:36.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:16:36.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T11:16:36.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:16:36.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:16:36.091 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:16:36.091 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T11:16:36.091 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:16:36.091 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:16:36.092 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:36.092 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:36.092 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:16:36.092 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:16:36.092 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:16:36.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:16:36.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:16:36.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:16:36.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T11:16:36.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:16:36.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:16:36.187 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:36.187 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:36.187 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:16:36.187 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:16:36.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:16:36.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-09-18T11:16:36.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:16:36.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj75 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-09-18T11:16:36.284 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:16:36.815 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:16:36.816 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:16:36.816 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:16:36.816 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:16:36.816 INFO:tasks.workunit.client.0.smithi174.stderr:2147483647' 2024-09-18T11:16:36.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-09-18T11:16:36.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:16:36.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T11:16:36.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:16:36.817 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-18T11:16:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:16:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T11:16:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:16:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T11:16:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:16:36.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:16:36.821 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:16:36.821 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T11:16:36.821 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:16:36.821 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:16:36.822 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:36.822 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:36.822 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:16:36.822 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:16:36.822 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:16:36.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:16:36.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:16:36.915 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T11:16:36.916 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:16:36.916 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:16:36.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:16:36.916 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:16:36.916 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:36.917 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:16:36.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:16:36.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:16:36.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-09-18T11:16:37.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-09-18T11:16:37.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:16:37.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:16:37.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:16:37.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:16:37.009 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:16:37.010 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:16:37.010 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:16:37.010 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:16:37.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:16:37.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:16:37.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:16:37.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:16:37.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:16:37.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:16:37.012 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:16:37.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:16:37.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:16:37.013 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:16:37.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:16:37.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:16:37.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:16:37.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:16:37.037 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:16:37.032+0000 7f9941f68540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:37.040 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:16:37.036+0000 7f9941f68540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:37.042 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:16:37.036+0000 7f9941f68540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:16:37.407 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:16:37.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:37.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:16:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:16:37.576+0000 7f9941f68540 -1 Falling back to public interface 2024-09-18T11:16:37.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:38.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:38.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:38.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:16:38.823 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:16:38.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:38.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:16:39.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:40.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:40.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:40.240 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:16:40.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:16:40.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:40.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:16:40.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:16:41.075 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:16:41.072+0000 7f9941f68540 -1 osd.2 47 log_to_monitors true 2024-09-18T11:16:41.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:16:41.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:16:41.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:16:41.678 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:16:41.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:16:41.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:16:42.154 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up out weight 0 up_from 54 up_thru 21 down_at 48 last_clean_interval [20,47) [v2:127.0.0.1:6818/178300,v1:127.0.0.1:6819/178300] [v2:127.0.0.1:6820/178300,v1:127.0.0.1:6821/178300] exists,up d5a24a5a-8e39-49b1-b462-b3fe026fc12c 2024-09-18T11:16:42.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:16:42.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:16:42.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:16:42.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-09-18T11:16:43.454 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-18T11:16:43.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-09-18T11:16:58.469 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-09-18T11:16:58.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-09-18T11:16:58.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-09-18T11:16:58.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-09-18T11:16:58.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-09-18T11:16:58.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-09-18T11:16:58.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-18T11:16:58.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-18T11:16:58.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-18T11:16:58.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-09-18T11:16:58.874 INFO:tasks.workunit.client.0.smithi174.stdout:active+recovery_unfound+degraded 2024-09-18T11:16:58.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-09-18T11:16:58.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-09-18T11:16:58.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-18T11:16:47.237231+0000 52'101 58:346 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-09-18T11:15:56.962778+0000 0'0 2024-09-18T11:15:56.962778+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:47:22.797340+0000 0 0 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:16:44.338756+0000 0'0 58:52 [4,1,2] 4 [4,1,2] 4 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T19:37:48.079693+0000 0 0 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.277336+0000 0'0 56:116 [0,4,1] 0 [0,4,1] 0 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T17:40:14.216542+0000 0 0 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:15:56.570254+0000 0'0 57:25 [4,3,0] 4 [4,3,0] 4 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T15:07:02.049217+0000 0 0 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-18T11:16:44.338286+0000 0'0 58:79 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-18T11:12:50.424039+0000 0'0 2024-09-18T11:12:50.424039+0000 0 0 periodic scrub scheduled @ 2024-09-19T14:52:18.481146+0000 0 0 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout: 2024-09-18T11:16:59.232 INFO:tasks.workunit.client.0.smithi174.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-18T11:16:59.234 INFO:tasks.workunit.client.0.smithi174.stderr:dumped pgs 2024-09-18T11:16:59.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-18T11:16:59.365 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-18T11:16:59.365 INFO:tasks.workunit.client.0.smithi174.stdout: "num_missing": 1, 2024-09-18T11:16:59.365 INFO:tasks.workunit.client.0.smithi174.stdout: "num_unfound": 1, 2024-09-18T11:16:59.365 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": [ 2024-09-18T11:16:59.365 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.365 INFO:tasks.workunit.client.0.smithi174.stdout: "oid": { 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "oid": "obj75", 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "key": "", 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "snapid": -2, 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "hash": 3565005477, 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "max": 0, 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "pool": 2, 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "namespace": "" 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "need": "52'76", 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "have": "0'0", 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "flags": "none", 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "locations": [ 2024-09-18T11:16:59.366 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: ] 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "NotRecovering", 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: "available_might_have_unfound": true, 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: "might_have_unfound": [], 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout: "more": false 2024-09-18T11:16:59.367 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-18T11:16:59.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-09-18T11:16:59.519 INFO:tasks.workunit.client.0.smithi174.stdout:{ 2024-09-18T11:16:59.519 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trimq": "[]", 2024-09-18T11:16:59.519 INFO:tasks.workunit.client.0.smithi174.stdout: "snap_trimq_len": 0, 2024-09-18T11:16:59.519 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+recovery_unfound+degraded", 2024-09-18T11:16:59.519 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch": 58, 2024-09-18T11:16:59.519 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.520 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_recovery_backfill": [ 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "2(4)", 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.521 INFO:tasks.workunit.client.0.smithi174.stdout: "info": { 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s0", 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "52'101", 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "52'75", 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "0'0", 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 101, 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 45, 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 45, 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 56, 2024-09-18T11:16:59.522 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 45, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 56, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 56, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 45, 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.523 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "52'101", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 347, 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 58, 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+recovery_unfound+degraded", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T11:16:47.306782+0000", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T11:16:47.237231+0000", 2024-09-18T11:16:59.524 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T11:16:47.306782+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T11:16:47.306782+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T11:16:44.387082+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T11:16:44.387082+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T11:16:47.306782+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T11:16:43.929518+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T11:16:47.306782+0000", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 56, 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "0'0", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "0'0", 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 45, 2024-09-18T11:16:59.525 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.526 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 0, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T11:16:59.527 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T17:47:22.797340+0000", 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 413696, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-18T11:16:59.528 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 505, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 1, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 3, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 1, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T11:16:59.529 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 404, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 99, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 405504, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T11:16:59.530 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T11:16:59.531 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.532 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-09-18T11:16:59.533 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 100 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),4(2)", 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 1 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.534 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T11:16:59.535 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.536 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.536 INFO:tasks.workunit.client.0.smithi174.stdout: "peer_info": [ 2024-09-18T11:16:59.536 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.536 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "0(3)", 2024-09-18T11:16:59.536 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s3", 2024-09-18T11:16:59.536 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "52'101", 2024-09-18T11:16:59.537 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "52'101", 2024-09-18T11:16:59.537 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "0'0", 2024-09-18T11:16:59.537 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 101, 2024-09-18T11:16:59.537 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T11:16:59.537 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T11:16:59.537 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T11:16:59.538 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 45, 2024-09-18T11:16:59.538 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 45, 2024-09-18T11:16:59.538 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.538 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 56, 2024-09-18T11:16:59.538 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.538 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 45, 2024-09-18T11:16:59.539 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T11:16:59.539 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T11:16:59.539 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 56, 2024-09-18T11:16:59.539 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 56, 2024-09-18T11:16:59.539 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 45, 2024-09-18T11:16:59.539 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.540 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.540 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.540 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.540 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.540 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T11:16:59.540 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.541 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T11:16:59.541 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "52'101", 2024-09-18T11:16:59.541 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 128, 2024-09-18T11:16:59.541 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 52, 2024-09-18T11:16:59.541 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-18T11:16:59.541 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.542 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.542 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.542 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.542 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.542 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.542 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.543 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.543 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T11:16:07.427045+0000", 2024-09-18T11:16:59.543 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T11:16:07.426547+0000", 2024-09-18T11:16:59.543 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 56, 2024-09-18T11:16:59.543 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "0'0", 2024-09-18T11:16:59.543 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "0'0", 2024-09-18T11:16:59.544 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 45, 2024-09-18T11:16:59.544 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.544 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T11:16:59.544 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T11:16:59.544 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.544 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.545 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.545 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.545 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.545 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T11:16:59.545 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T11:16:59.545 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 0, 2024-09-18T11:16:59.546 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T11:16:59.546 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T11:16:59.546 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T11:16:59.546 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T11:16:59.546 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T11:16:59.546 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T11:16:59.547 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T11:16:59.547 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T11:16:59.547 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T11:16:59.547 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T11:16:59.547 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T15:31:32.653512+0000", 2024-09-18T11:16:59.548 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T11:16:59.548 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T11:16:59.548 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T11:16:59.548 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T11:16:59.548 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 413696, 2024-09-18T11:16:59.548 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-18T11:16:59.549 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T11:16:59.549 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 505, 2024-09-18T11:16:59.549 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:16:59.549 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-18T11:16:59.549 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T11:16:59.550 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 404, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T11:16:59.551 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T11:16:59.552 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T11:16:59.552 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:16:59.552 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:16:59.552 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T11:16:59.552 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T11:16:59.552 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T11:16:59.553 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.554 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T11:16:59.555 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 101 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.556 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "1(1)", 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s1", 2024-09-18T11:16:59.557 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "52'101", 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "52'101", 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "0'0", 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 101, 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 45, 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 45, 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.558 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 56, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 45, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 56, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 56, 2024-09-18T11:16:59.559 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 45, 2024-09-18T11:16:59.560 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.560 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.560 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.560 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.560 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.561 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T11:16:59.561 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.561 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T11:16:59.561 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "52'101", 2024-09-18T11:16:59.561 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 128, 2024-09-18T11:16:59.561 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 52, 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.562 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T11:16:07.427045+0000", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T11:16:07.426547+0000", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 56, 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "0'0", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "0'0", 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 45, 2024-09-18T11:16:59.563 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 0, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T11:16:59.564 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T15:31:32.653512+0000", 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 413696, 2024-09-18T11:16:59.565 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 505, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 1, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 404, 2024-09-18T11:16:59.566 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T11:16:59.567 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T11:16:59.568 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.569 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 101 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.570 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.571 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "2(4)", 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s4", 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "52'101", 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "47'1", 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "0'0", 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 1, 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 45, 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 45, 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 56, 2024-09-18T11:16:59.572 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 45, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 56, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 56, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 45, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.573 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "47'1", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 14, 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 47, 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+clean", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T11:15:59.000275+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T11:15:58.999115+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T11:15:58.999115+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.574 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 56, 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "0'0", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "0'0", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 45, 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.575 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 1, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 0, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 1, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T11:16:59.576 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T15:46:55.402099+0000", 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 4096, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 1, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 5, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 1, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 0, 2024-09-18T11:16:59.577 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 1, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 1, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 4, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T11:16:59.578 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:16:59.579 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T11:16:59.580 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [], 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [], 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.581 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "peer": "4(2)", 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "pgid": "2.0s2", 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "last_update": "52'101", 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "last_complete": "52'101", 2024-09-18T11:16:59.582 INFO:tasks.workunit.client.0.smithi174.stdout: "log_tail": "0'0", 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_user_version": 101, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill": "MAX", 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [], 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "history": { 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_created": 45, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "epoch_pool_created": 45, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_started": 56, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_interval_clean": 45, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_split": 0, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_marked_full": 0, 2024-09-18T11:16:59.583 INFO:tasks.workunit.client.0.smithi174.stdout: "same_up_since": 56, 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "same_interval_since": 56, 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "same_primary_since": 45, 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "prior_readable_until_ub": 0 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "stats": { 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "version": "52'101", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_seq": 128, 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "reported_epoch": 52, 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "state": "active+undersized+degraded", 2024-09-18T11:16:59.584 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fresh": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_change": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_active": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_peered": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean": "2024-09-18T11:16:05.781158+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_active": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_became_peered": "2024-09-18T11:16:07.875476+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_unstale": "2024-09-18T11:16:35.656873+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_undegraded": "2024-09-18T11:16:07.427045+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_fullsized": "2024-09-18T11:16:07.426547+0000", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "mapping_epoch": 56, 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "log_start": "0'0", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_start": "0'0", 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "created": 45, 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_clean": 46, 2024-09-18T11:16:59.585 INFO:tasks.workunit.client.0.smithi174.stdout: "parent": "0.0", 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "parent_split_bits": 0, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub": "0'0", 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub": "0'0", 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "last_deep_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "last_clean_scrub_stamp": "2024-09-18T11:15:56.962778+0000", 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_scrubbed": 0, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "log_size": 101, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "log_dups_size": 0, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "ondisk_log_size": 101, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "stats_invalid": false, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "dirty_stats_invalid": false, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "omap_stats_invalid": false, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_stats_invalid": false, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "hitset_bytes_stats_invalid": false, 2024-09-18T11:16:59.586 INFO:tasks.workunit.client.0.smithi174.stdout: "pin_stats_invalid": false, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "manifest_stats_invalid": false, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrimq_len": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "last_scrub_duration": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-19T15:31:32.653512+0000", 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_duration": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "objects_trimmed": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "snaptrim_duration": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "stat_sum": { 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes": 413696, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects": 101, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_clones": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_object_copies": 505, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing_on_primary": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_missing": 0, 2024-09-18T11:16:59.587 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_degraded": 101, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_misplaced": 0, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_unfound": 0, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_dirty": 101, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_whiteouts": 0, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read": 0, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_read_kb": 0, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write": 101, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_write_kb": 404, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_scrub_errors": 0, 2024-09-18T11:16:59.588 INFO:tasks.workunit.client.0.smithi174.stdout: "num_shallow_scrub_errors": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_deep_scrub_errors": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_recovered": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_recovered": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_keys_recovered": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_omap": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_hit_set_archive": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_bytes_hit_set_archive": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_kb": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_kb": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_promote": 0, 2024-09-18T11:16:59.589 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_high": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_flush_mode_low": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_some": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_evict_mode_full": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_pinned": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_legacy_snapsets": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_large_omap_objects": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_manifest": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_bytes": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_omap_keys": 0, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "num_objects_repaired": 0 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: "up": [ 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.590 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: "acting": [ 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 3, 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 1, 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 4, 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 0, 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: 2 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: "avail_no_missing": [ 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: "3(0)", 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: "0(3)", 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: "1(1)", 2024-09-18T11:16:59.591 INFO:tasks.workunit.client.0.smithi174.stdout: "4(2)" 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "object_location_counts": [ 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": 101 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "blocked_by": [], 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "up_primary": 3, 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "acting_primary": 3, 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "purged_snaps": [] 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "empty": 0, 2024-09-18T11:16:59.592 INFO:tasks.workunit.client.0.smithi174.stdout: "dne": 0, 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "incomplete": 0, 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "last_epoch_started": 58, 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "hit_set_history": { 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "current_last_update": "0'0", 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "history": [] 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_state": [ 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "Started/Primary/Active", 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "enter_time": "2024-09-18T11:16:43.927001+0000", 2024-09-18T11:16:59.593 INFO:tasks.workunit.client.0.smithi174.stdout: "might_have_unfound": [ 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "0(3)", 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "1(1)", 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "2(4)", 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.594 INFO:tasks.workunit.client.0.smithi174.stdout: "osd": "4(2)", 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "status": "already probed" 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_progress": { 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "backfill_targets": [], 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "waiting_on_backfill": [], 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "last_backfill_started": "MIN", 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "backfill_info": { 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "begin": "MIN", 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "end": "MIN", 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "objects": [] 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "peer_backfill_info": [], 2024-09-18T11:16:59.595 INFO:tasks.workunit.client.0.smithi174.stdout: "backfills_in_flight": [], 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "recovering": [], 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "pg_backend": { 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "recovery_ops": [], 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "read_ops": [] 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: { 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "name": "Started", 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "enter_time": "2024-09-18T11:16:43.024708+0000" 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: } 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: ], 2024-09-18T11:16:59.596 INFO:tasks.workunit.client.0.smithi174.stdout: "scrubber": { 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "active": false, 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "must_scrub": false, 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "must_deep_scrub": false, 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "must_repair": false, 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "need_auto": false, 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "scrub_reg_stamp": "2024-09-19T17:47:22.797340+0000", 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "schedule": "scrub scheduled @ 2024-09-19T17:47:22.797340+0000" 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: }, 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout: "agent_state": {} 2024-09-18T11:16:59.597 INFO:tasks.workunit.client.0.smithi174.stdout:} 2024-09-18T11:16:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-18T11:16:59.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-09-18T11:16:59.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-18T11:16:59.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-09-18T11:16:59.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-09-18T11:16:59.814 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-09-18T11:16:59.815 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-18T11:16:59.815 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-09-18T11:16:59.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-09-18T11:16:59.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-09-18T11:16:59.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-18T11:17:04.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-09-18T11:17:04.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-18T11:17:05.241 INFO:tasks.workunit.client.0.smithi174.stderr:pg has no unfound objects 2024-09-18T11:17:05.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-09-18T11:17:05.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:17:05.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:17:05.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:17:05.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:17:05.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:17:05.254 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:17:05.254 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:17:05.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:17:05.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:17:05.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:17:05.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:17:05.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:17:05.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:17:05.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:17:05.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:17:05.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:17:05.878 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:17:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:17:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:17:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:4' 2024-09-18T11:17:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:17:05.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:17:05.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:17:06.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803833 2024-09-18T11:17:06.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803833 2024-09-18T11:17:06.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833' 2024-09-18T11:17:06.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:17:06.046 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:17:06.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542191 2024-09-18T11:17:06.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542191 2024-09-18T11:17:06.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542191' 2024-09-18T11:17:06.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:17:06.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:17:06.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=231928233991 2024-09-18T11:17:06.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 231928233991 2024-09-18T11:17:06.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542191 2-231928233991' 2024-09-18T11:17:06.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:17:06.347 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:17:06.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051620 2024-09-18T11:17:06.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051620 2024-09-18T11:17:06.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542191 2-231928233991 3-124554051620' 2024-09-18T11:17:06.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:17:06.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:17:06.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724576 2024-09-18T11:17:06.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724576 2024-09-18T11:17:06.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803833 1-60129542191 2-231928233991 3-124554051620 4-167503724576' 2024-09-18T11:17:06.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:17:06.643 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803833 2024-09-18T11:17:06.643 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:17:06.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:17:06.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803833 2024-09-18T11:17:06.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:17:06.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803833 2024-09-18T11:17:06.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-09-18T11:17:06.648 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803833 2024-09-18T11:17:06.649 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:17:07.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803830 -lt 25769803833 2024-09-18T11:17:07.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:17:08.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:17:08.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:17:08.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803833 -lt 25769803833 2024-09-18T11:17:08.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:17:08.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542191 2024-09-18T11:17:08.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:17:08.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:17:08.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542191 2024-09-18T11:17:08.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:17:08.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542191 2024-09-18T11:17:08.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542191' 2024-09-18T11:17:08.502 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 60129542191 2024-09-18T11:17:08.502 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:17:08.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542192 -lt 60129542191 2024-09-18T11:17:08.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:17:08.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-231928233991 2024-09-18T11:17:08.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:17:08.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:17:08.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-231928233991 2024-09-18T11:17:08.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:17:08.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=231928233991 2024-09-18T11:17:08.942 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 231928233991 2024-09-18T11:17:08.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 231928233991' 2024-09-18T11:17:08.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:17:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 231928233992 -lt 231928233991 2024-09-18T11:17:09.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:17:09.400 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051620 2024-09-18T11:17:09.400 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:17:09.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:17:09.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051620 2024-09-18T11:17:09.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:17:09.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051620 2024-09-18T11:17:09.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051620' 2024-09-18T11:17:09.406 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051620 2024-09-18T11:17:09.406 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:17:09.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051621 -lt 124554051620 2024-09-18T11:17:09.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:17:09.836 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-167503724576 2024-09-18T11:17:09.836 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:17:09.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:17:09.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-167503724576 2024-09-18T11:17:09.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:17:09.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724576 2024-09-18T11:17:09.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 167503724576' 2024-09-18T11:17:09.842 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 167503724576 2024-09-18T11:17:09.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:17:10.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724576 -lt 167503724576 2024-09-18T11:17:10.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:17:10.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:17:10.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:17:10.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:17:10.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:17:10.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:17:10.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:17:10.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:17:10.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:17:10.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:17:10.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:17:11.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:17:11.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:17:11.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:17:11.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:17:11.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:17:11.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:17:11.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:17:11.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-09-18T11:17:11.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:11.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-18T11:17:11.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-09-18T11:17:11.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:11.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:11.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-18T11:17:11.829 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-09-18T11:17:11.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-18T11:17:11.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-09-18T11:17:11.975 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:11.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:11.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-18T11:17:11.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-18T11:17:12.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-18T11:17:12.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-18T11:17:12.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-18T11:17:12.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-18T11:17:12.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-18T11:17:12.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-18T11:17:12.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-18T11:17:12.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-18T11:17:12.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-18T11:17:12.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-18T11:17:12.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-18T11:17:12.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-18T11:17:12.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-18T11:17:12.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.631 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-18T11:17:12.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-18T11:17:12.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-18T11:17:12.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-18T11:17:12.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-18T11:17:12.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-18T11:17:12.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-18T11:17:12.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-18T11:17:12.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-09-18T11:17:12.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:12.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:12.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-18T11:17:12.981 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-18T11:17:13.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-18T11:17:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-18T11:17:13.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-18T11:17:13.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-18T11:17:13.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.245 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-18T11:17:13.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-18T11:17:13.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-18T11:17:13.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-18T11:17:13.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-18T11:17:13.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-18T11:17:13.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-18T11:17:13.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-18T11:17:13.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-18T11:17:13.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-18T11:17:13.632 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-18T11:17:13.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-18T11:17:13.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-18T11:17:13.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-18T11:17:13.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-18T11:17:13.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-18T11:17:13.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-18T11:17:13.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-09-18T11:17:13.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:13.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:13.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-18T11:17:13.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-18T11:17:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-18T11:17:14.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-18T11:17:14.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-18T11:17:14.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-18T11:17:14.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-18T11:17:14.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-18T11:17:14.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-18T11:17:14.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-18T11:17:14.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-18T11:17:14.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-18T11:17:14.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-18T11:17:14.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-18T11:17:14.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-18T11:17:14.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-18T11:17:14.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-18T11:17:14.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-18T11:17:14.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-18T11:17:14.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-18T11:17:14.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-18T11:17:14.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-18T11:17:14.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-18T11:17:14.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-09-18T11:17:14.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:14.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:14.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-18T11:17:14.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-18T11:17:15.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-18T11:17:15.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.111 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-09-18T11:17:15.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-18T11:17:15.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-18T11:17:15.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-18T11:17:15.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-18T11:17:15.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-18T11:17:15.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-18T11:17:15.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-18T11:17:15.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-18T11:17:15.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-18T11:17:15.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-18T11:17:15.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-18T11:17:15.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-18T11:17:15.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-18T11:17:15.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-18T11:17:15.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-18T11:17:15.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-18T11:17:15.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-18T11:17:15.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-18T11:17:15.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-18T11:17:15.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-18T11:17:15.938 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-09-18T11:17:15.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:15.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:15.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-18T11:17:15.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-09-18T11:17:16.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:16.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:16.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-18T11:17:16.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-09-18T11:17:16.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:16.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:16.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-18T11:17:16.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-09-18T11:17:16.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:16.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:16.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-18T11:17:16.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-09-18T11:17:16.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:16.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-18T11:17:16.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-18T11:17:16.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-09-18T11:17:16.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:16.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-09-18T11:17:16.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:17:16.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:17:16.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:17:16.940 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:17:16.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:17:18.070 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:17:18.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:17:18.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:17:18.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:18.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:17:18.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:18.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:18.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:18.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:18.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:18.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:18.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:18.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:18.205 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:18.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:18.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:18.207 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:18.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:18.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:18.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:18.210 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:18.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:18.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:18.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:17:18.252 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:18.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:18.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:18.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:17:18.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:17:18.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:17:18.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:17:18.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:17:18.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:17:18.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:17:18.260 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:17:18.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:17:18.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:17:18.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:17:18.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:17:18.264 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:17:18.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:18.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:17:18.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:17:18.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:17:18.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:17:18.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:17:18.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:17:18.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:17:18.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:18.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:18.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:17:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:17:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:17:18.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:17:18.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:17:18.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:18.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:18.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:17:18.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:17:18.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:17:18.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:17:18.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:17:18.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:17:18.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:17:18.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:17:18.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:17:18.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:17:18.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:17:18.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:17:18.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:17:18.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:18.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:18.651 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:18.651 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:18.651 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:18.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:17:18.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:17:18.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:17:18.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:17:18.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:17:18.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:17:18.982 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:17:18.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:17:18.983 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:17:18.983 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:18.983 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:18.984 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:18.985 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:18.985 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:18.985 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:17:18.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:17:18.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:17:19.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:17:19.078 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:17:19.078 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:17:19.078 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:17:19.079 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:19.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:17:19.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:17:19.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:17:19.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:17:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:17:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:17:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:17:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:17:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:17:19.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:17:20.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:17:20.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:20.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:20.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:20.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:20.067 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:20.067 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:17:20.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:17:20.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:17:20.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:17:20.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:17:20.419 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:17:20.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:17:21.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:17:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:17:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:17:21.438 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:17:21.438 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:21.438 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:21.438 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:17:21.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:17:21.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:17:21.513 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:17:21.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:17:21.523 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:17:18.691+0000 7f765a330a00 0 load: jerasure load: lrc 2024-09-18T11:17:21.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-09-18T11:17:21.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-09-18T11:17:21.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-09-18T11:17:21.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-09-18T11:17:21.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-09-18T11:17:21.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:17:21.524 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-09-18T11:17:21.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-09-18T11:17:21.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:17:21.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:17:21.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:17:21.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:17:21.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:17:21.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:17:21.531 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:17:21.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d93d4305-0d25-4fd4-877f-5fd26e348dbb 2024-09-18T11:17:21.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d93d4305-0d25-4fd4-877f-5fd26e348dbb' 2024-09-18T11:17:21.533 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 d93d4305-0d25-4fd4-877f-5fd26e348dbb 2024-09-18T11:17:21.533 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:17:21.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBtupm8jn+IBAAGBEnL8+kMwQSBiz8JCBgZQ== 2024-09-18T11:17:21.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBtupm8jn+IBAAGBEnL8+kMwQSBiz8JCBgZQ=="}' 2024-09-18T11:17:21.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d93d4305-0d25-4fd4-877f-5fd26e348dbb -i td/test-erasure-eio/0/new.json 2024-09-18T11:17:21.909 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:17:21.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:17:21.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBtupm8jn+IBAAGBEnL8+kMwQSBiz8JCBgZQ== --osd-uuid d93d4305-0d25-4fd4-877f-5fd26e348dbb 2024-09-18T11:17:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:21.947+0000 7fa635c92540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:21.957 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:21.951+0000 7fa635c92540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:21.958 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:21.951+0000 7fa635c92540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:21.959 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:21.955+0000 7fa635c92540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:17:21.959 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:21.955+0000 7fa635c92540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:17:21.959 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:21.955+0000 7fa635c92540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:17:22.055 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:22.051+0000 7fa635c92540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:17:28.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:17:28.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:17:28.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:17:28.098 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:17:28.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:17:28.649 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:17:28.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:17:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:17:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:17:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:17:28.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:17:28.679 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:28.675+0000 7fb536964540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:28.682 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:28.675+0000 7fb536964540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:28.684 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:28.679+0000 7fb536964540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:29.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:17:29.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:17:29.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:17:29.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:17:29.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:17:29.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:29.034 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:17:29.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:17:29.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:29.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:29.245 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:29.239+0000 7fb536964540 -1 Falling back to public interface 2024-09-18T11:17:29.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:30.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:30.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:30.464 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:17:30.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:17:30.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:30.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:30.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:31.897 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:31.891+0000 7fb536964540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:17:31.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:31.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:31.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:17:31.920 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:17:31.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:31.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:32.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:33.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:33.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:33.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:17:33.447 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:17:33.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:33.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:33.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:34.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:34.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:34.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:17:34.918 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:17:34.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:34.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:35.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:36.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:36.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:36.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:17:36.373 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:17:36.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:36.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:36.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:37.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:37.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:37.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:17:37.895 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:17:37.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:37.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:38.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:39.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:39.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:39.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:17:39.318 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:17:39.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:39.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:39.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:40.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:40.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:40.785 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:17:40.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:17:40.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:40.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:41.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:42.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:42.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:42.238 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:17:42.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:17:42.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:42.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:42.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:17:43.810 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:17:43.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:43.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:44.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:45.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:45.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:45.436 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:17:45.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:17:45.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:45.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:46.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:47.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:47.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:47.058 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:17:47.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:17:47.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:47.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:47.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:48.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:48.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:48.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:17:48.543 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:17:48.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:48.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:50.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:50.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:50.660 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:17:50.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:17:50.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:50.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:51.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:52.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:52.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:52.672 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:17:52.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:17:52.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:52.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:53.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:54.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:54.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:54.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:17:54.404 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:17:54.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:54.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:54.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:17:55.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:17:55.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:17:55.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:17:55.951 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:17:55.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:17:55.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:17:56.369 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/182961,v1:127.0.0.1:6803/182961] [v2:127.0.0.1:6804/182961,v1:127.0.0.1:6805/182961] exists,up d93d4305-0d25-4fd4-877f-5fd26e348dbb 2024-09-18T11:17:56.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:17:56.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:17:56.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:17:56.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:17:56.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:17:56.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:17:56.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:17:56.373 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:17:56.373 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:17:56.373 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:17:56.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:17:56.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:17:56.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:17:56.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:17:56.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:17:56.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:17:56.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:17:56.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f8ff04e6-e8ed-4a30-a6be-923844e0b0af 2024-09-18T11:17:56.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f8ff04e6-e8ed-4a30-a6be-923844e0b0af' 2024-09-18T11:17:56.378 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 f8ff04e6-e8ed-4a30-a6be-923844e0b0af 2024-09-18T11:17:56.378 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:17:56.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDktupmbAU7FxAANWB1Sf7aOEP5YCLtb4JIyQ== 2024-09-18T11:17:56.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDktupmbAU7FxAANWB1Sf7aOEP5YCLtb4JIyQ=="}' 2024-09-18T11:17:56.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f8ff04e6-e8ed-4a30-a6be-923844e0b0af -i td/test-erasure-eio/1/new.json 2024-09-18T11:17:56.929 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:17:56.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:17:56.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDktupmbAU7FxAANWB1Sf7aOEP5YCLtb4JIyQ== --osd-uuid f8ff04e6-e8ed-4a30-a6be-923844e0b0af 2024-09-18T11:17:56.973 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:56.967+0000 7f30be1d2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:56.977 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:56.971+0000 7f30be1d2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:56.979 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:56.975+0000 7f30be1d2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:17:56.981 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:56.975+0000 7f30be1d2540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:17:56.981 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:56.975+0000 7f30be1d2540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:17:56.981 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:56.975+0000 7f30be1d2540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:17:57.022 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:17:57.015+0000 7f30be1d2540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:18:03.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:18:03.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:18:03.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:18:03.610 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:18:03.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:18:04.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:18:04.140 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:18:04.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:18:04.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:18:04.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:18:04.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:18:04.170 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:04.163+0000 7fcd49871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:04.172 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:04.167+0000 7fcd49871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:04.178 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:04.167+0000 7fcd49871540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:04.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:18:04.532 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:18:04.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:04.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:04.737 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:04.731+0000 7fcd49871540 -1 Falling back to public interface 2024-09-18T11:18:04.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:05.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:05.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:05.946 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:18:05.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:18:05.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:05.946 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:06.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:07.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:07.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:07.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:18:07.388 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:18:07.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:07.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:07.481 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:07.475+0000 7fcd49871540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:18:07.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:08.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:08.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:08.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:18:08.800 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:18:08.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:08.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:09.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:10.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:10.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:10.235 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:18:10.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:18:10.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:10.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:10.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:11.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:11.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:11.726 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:18:11.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:18:11.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:11.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:12.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:13.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:13.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:13.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:18:13.272 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:18:13.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:13.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:13.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:14.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:14.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:14.744 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:18:14.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:18:14.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:14.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:15.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:16.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:16.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:16.280 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:18:16.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:18:16.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:16.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:16.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:17.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:17.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:17.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:18:17.781 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:18:17.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:17.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:18.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:19.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:19.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:19.400 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:18:19.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:18:19.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:19.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:19.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:20.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:20.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:20.830 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:18:20.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:18:20.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:20.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:21.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:22.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:22.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:22.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:18:22.415 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:18:22.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:22.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:23.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:24.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:24.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:24.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:18:24.274 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:18:24.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:24.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:24.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:25.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:25.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:25.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:18:25.914 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:18:25.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:25.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:27.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:27.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:27.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:18:27.997 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:18:27.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:27.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:28.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:29.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:29.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:29.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:18:29.640 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:18:29.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:29.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:30.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:31.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:31.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:31.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:18:31.075 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:18:31.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:31.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:31.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:32.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:32.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:32.706 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:18:32.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:18:32.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:32.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:18:33.127 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/184737,v1:127.0.0.1:6811/184737] [v2:127.0.0.1:6812/184737,v1:127.0.0.1:6813/184737] exists,up f8ff04e6-e8ed-4a30-a6be-923844e0b0af 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:18:33.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:18:33.130 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:18:33.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:18:33.132 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:18:33.132 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:18:33.132 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:18:33.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:18:33.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:18:33.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:18:33.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9b2acf07-c7ac-4d0f-b17e-edac1fb310cb 2024-09-18T11:18:33.135 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 9b2acf07-c7ac-4d0f-b17e-edac1fb310cb 2024-09-18T11:18:33.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9b2acf07-c7ac-4d0f-b17e-edac1fb310cb' 2024-09-18T11:18:33.136 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:18:33.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJt+pmE1UoCRAAcKllwthPz2nwxbRczhXrHw== 2024-09-18T11:18:33.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJt+pmE1UoCRAAcKllwthPz2nwxbRczhXrHw=="}' 2024-09-18T11:18:33.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9b2acf07-c7ac-4d0f-b17e-edac1fb310cb -i td/test-erasure-eio/2/new.json 2024-09-18T11:18:33.728 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:18:33.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:18:33.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJt+pmE1UoCRAAcKllwthPz2nwxbRczhXrHw== --osd-uuid 9b2acf07-c7ac-4d0f-b17e-edac1fb310cb 2024-09-18T11:18:33.778 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.770+0000 7fe8ef7d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:33.781 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.774+0000 7fe8ef7d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:33.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.778+0000 7fe8ef7d3540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:33.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.778+0000 7fe8ef7d3540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:18:33.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.778+0000 7fe8ef7d3540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:18:33.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.778+0000 7fe8ef7d3540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:18:33.892 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:33.886+0000 7fe8ef7d3540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:18:40.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:18:40.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:18:40.370 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:18:40.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:18:40.370 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:18:40.906 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:18:40.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:18:40.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:18:40.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:18:40.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:18:40.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:18:40.935 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:40.926+0000 7f032454b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:40.937 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:40.930+0000 7f032454b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:40.938 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:40.930+0000 7f032454b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:41.451 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:18:41.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:18:41.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:41.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:41.517 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:41.510+0000 7f032454b540 -1 Falling back to public interface 2024-09-18T11:18:41.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:42.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:42.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:42.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:18:42.873 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:18:42.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:42.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:43.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:44.133 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:18:44.126+0000 7f032454b540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:18:44.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:44.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:44.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:18:44.280 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:18:44.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:44.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:44.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:45.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:45.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:45.812 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:18:45.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:18:45.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:45.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:46.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:47.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:47.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:47.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:18:47.288 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:18:47.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:47.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:47.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:48.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:48.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:48.719 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:18:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:18:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:49.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:50.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:50.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:50.199 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:18:50.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:18:50.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:50.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:50.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:51.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:51.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:51.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:18:51.760 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:18:51.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:51.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:52.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:53.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:53.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:53.453 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:18:53.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:18:53.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:53.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:54.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:55.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:55.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:55.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:18:55.043 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:18:55.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:55.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:55.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:56.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:56.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:56.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:18:56.511 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:18:56.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:56.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:56.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:57.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:57.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:57.969 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:18:57.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:18:57.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:57.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:18:58.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:18:59.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:18:59.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:18:59.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:18:59.437 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:18:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:18:59.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:19:00.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:01.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:01.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:01.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:19:01.352 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:19:01.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:01.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:19:02.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:03.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:03.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:03.096 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:19:03.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:19:03.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:03.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:19:03.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:04.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:04.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:04.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:19:04.650 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:19:04.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:04.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:19:05.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:06.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:06.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:06.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:19:06.877 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:19:06.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:06.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:19:07.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:08.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:08.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:08.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:19:08.760 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:19:08.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:08.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:19:09.348 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/186538,v1:127.0.0.1:6819/186538] [v2:127.0.0.1:6820/186538,v1:127.0.0.1:6821/186538] exists,up 9b2acf07-c7ac-4d0f-b17e-edac1fb310cb 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:19:09.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:19:09.351 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:19:09.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:09.352 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:09.352 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:09.352 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:09.352 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:19:09.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:19:09.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:19:09.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:19:09.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:19:09.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:19:09.354 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:19:09.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5b932202-d1f8-4efc-bf32-9af80d582edc 2024-09-18T11:19:09.355 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 5b932202-d1f8-4efc-bf32-9af80d582edc 2024-09-18T11:19:09.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5b932202-d1f8-4efc-bf32-9af80d582edc' 2024-09-18T11:19:09.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:19:09.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtt+pmNUZ8FhAAQx7FKvmklOrlOj5VZdW94g== 2024-09-18T11:19:09.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtt+pmNUZ8FhAAQx7FKvmklOrlOj5VZdW94g=="}' 2024-09-18T11:19:09.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5b932202-d1f8-4efc-bf32-9af80d582edc -i td/test-erasure-eio/3/new.json 2024-09-18T11:19:10.053 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:19:10.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:19:10.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtt+pmNUZ8FhAAQx7FKvmklOrlOj5VZdW94g== --osd-uuid 5b932202-d1f8-4efc-bf32-9af80d582edc 2024-09-18T11:19:10.100 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.094+0000 7f795ebbc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:10.104 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.098+0000 7f795ebbc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:10.106 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.098+0000 7f795ebbc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.102+0000 7f795ebbc540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:19:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.102+0000 7f795ebbc540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:19:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.102+0000 7f795ebbc540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:19:10.225 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:10.218+0000 7f795ebbc540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:19:16.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:19:16.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:19:16.538 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:19:16.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:19:16.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:19:17.077 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:19:17.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:19:17.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:19:17.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:19:17.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:19:17.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:19:17.106 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:17.097+0000 7f1ff6ea0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:17.134 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:17.125+0000 7f1ff6ea0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:17.136 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:17.129+0000 7f1ff6ea0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:17.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:19:17.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:19:17.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:17.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:17.717 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:17.709+0000 7f1ff6ea0540 -1 Falling back to public interface 2024-09-18T11:19:17.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:18.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:18.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:19:18.961 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:19:18.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:18.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:19.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:20.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:20.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:20.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:19:20.348 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:19:20.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:20.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:20.523 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:20.517+0000 7f1ff6ea0540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:19:20.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:21.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:21.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:21.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:19:21.909 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:19:21.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:21.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:22.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:23.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:23.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:23.550 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:19:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:19:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:24.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:25.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:25.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:25.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:19:25.050 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:19:25.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:25.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:25.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:26.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:26.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:26.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:19:26.628 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:19:26.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:26.629 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:27.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:28.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:28.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:28.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:19:28.043 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:19:28.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:28.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:28.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:29.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:29.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:29.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:19:29.472 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:19:29.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:29.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:29.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:30.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:30.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:30.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:19:30.936 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:19:30.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:30.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:31.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:32.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:32.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:32.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:19:32.441 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:19:32.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:32.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:32.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:33.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:33.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:33.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:19:33.859 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:19:33.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:33.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:34.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:35.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:35.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:35.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:19:35.410 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:19:35.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:35.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:36.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:37.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:37.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:37.264 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:19:37.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:19:37.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:37.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:38.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:39.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:39.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:39.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:19:39.430 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:19:39.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:39.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:40.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:41.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:41.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:41.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:19:41.246 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:19:41.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:41.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:42.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:43.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:43.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:43.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:19:43.074 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:19:43.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:43.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:44.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:45.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:45.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:45.147 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:19:45.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:19:45.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:45.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:45.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:46.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:46.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:46.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:19:46.615 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:19:46.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:46.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:19:47.297 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/188313,v1:127.0.0.1:6827/188313] [v2:127.0.0.1:6828/188313,v1:127.0.0.1:6829/188313] exists,up 5b932202-d1f8-4efc-bf32-9af80d582edc 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:19:47.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:19:47.300 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:19:47.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:19:47.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:19:47.302 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:19:47.302 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:19:47.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:19:47.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:19:47.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-09-18T11:19:47.304 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:19:47.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=542e912f-8710-4d45-a77b-f26756454c57 2024-09-18T11:19:47.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 542e912f-8710-4d45-a77b-f26756454c57' 2024-09-18T11:19:47.305 INFO:tasks.workunit.client.0.smithi174.stdout:add osd4 542e912f-8710-4d45-a77b-f26756454c57 2024-09-18T11:19:47.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:19:47.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTt+pmmGRFExAAG1m2/KYoS7QTDd8wRdE/+Q== 2024-09-18T11:19:47.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTt+pmmGRFExAAG1m2/KYoS7QTDd8wRdE/+Q=="}' 2024-09-18T11:19:47.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 542e912f-8710-4d45-a77b-f26756454c57 -i td/test-erasure-eio/4/new.json 2024-09-18T11:19:48.021 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:19:48.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-09-18T11:19:48.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTt+pmmGRFExAAG1m2/KYoS7QTDd8wRdE/+Q== --osd-uuid 542e912f-8710-4d45-a77b-f26756454c57 2024-09-18T11:19:48.060 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.053+0000 7f7163cac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:48.064 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.057+0000 7f7163cac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:48.066 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.057+0000 7f7163cac540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:48.067 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.061+0000 7f7163cac540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:19:48.067 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.061+0000 7f7163cac540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:19:48.067 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.061+0000 7f7163cac540 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-09-18T11:19:48.202 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:48.193+0000 7f7163cac540 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-09-18T11:19:54.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-09-18T11:19:54.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:19:54.718 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd4 key to auth repository 2024-09-18T11:19:54.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-18T11:19:54.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:19:55.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-18T11:19:55.260 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.4 2024-09-18T11:19:55.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:19:55.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:19:55.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:19:55.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:19:55.282 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:55.273+0000 7fd5b5a39540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:55.285 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:55.277+0000 7fd5b5a39540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:55.286 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:55.277+0000 7fd5b5a39540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:19:55.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-18T11:19:55.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:19:55.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-18T11:19:55.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:19:55.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:19:55.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:55.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:19:55.742 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:19:55.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:55.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:19:55.848 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:55.841+0000 7fd5b5a39540 -1 Falling back to public interface 2024-09-18T11:19:56.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:57.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:57.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:57.140 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:19:57.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:19:57.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:57.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:19:57.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:19:58.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:19:58.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:19:58.575 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:19:58.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:19:58.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:19:58.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:19:58.732 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:19:58.725+0000 7fd5b5a39540 -1 osd.4 0 log_to_monitors true 2024-09-18T11:19:59.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:00.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:00.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:00.113 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:20:00.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:20:00.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:00.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:00.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:01.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:01.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:01.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:20:01.639 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:20:01.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:01.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:02.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:03.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:03.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:03.383 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:20:03.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:20:03.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:03.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:03.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:04.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:04.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:04.805 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:20:04.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:20:04.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:04.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:05.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:06.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:06.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:06.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:20:06.214 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:20:06.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:06.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:06.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:07.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:07.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:07.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:20:07.659 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:20:07.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:07.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:08.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:09.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:09.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:09.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:20:09.074 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:20:09.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:09.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:09.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:10.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:10.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:10.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:20:10.488 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:20:10.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:10.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:11.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:12.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:12.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:12.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:20:12.039 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:20:12.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:12.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:12.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:13.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:13.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:13.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:20:13.572 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:20:13.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:13.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:14.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:15.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:15.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:15.540 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:20:15.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:20:15.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:15.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:16.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:17.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:17.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:17.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:20:17.311 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:20:17.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:17.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:18.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:19.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:19.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:19.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:20:19.005 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:20:19.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:19.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:19.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:20.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:20.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:20:20.804 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:20:20.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:20.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:22.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:23.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:23.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:23.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:20:23.043 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:20:23.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:23.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:23.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:24.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:24.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:24.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:20:24.696 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:20:24.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:24.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-18T11:20:25.133 INFO:tasks.workunit.client.0.smithi174.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/190114,v1:127.0.0.1:6835/190114] [v2:127.0.0.1:6836/190114,v1:127.0.0.1:6837/190114] exists,up 542e912f-8710-4d45-a77b-f26756454c57 2024-09-18T11:20:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:20:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:20:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:20:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:20:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-09-18T11:20:25.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:20:25.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:20:25.137 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:20:25.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:20:25.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-09-18T11:20:25.140 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:20:25.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d856b86-98bc-4571-9567-b228ca0d10fc 2024-09-18T11:20:25.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 5d856b86-98bc-4571-9567-b228ca0d10fc' 2024-09-18T11:20:25.141 INFO:tasks.workunit.client.0.smithi174.stdout:add osd5 5d856b86-98bc-4571-9567-b228ca0d10fc 2024-09-18T11:20:25.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:20:25.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5t+pmqvd+CRAAFjRgg29SiVKugouHwdaK8Q== 2024-09-18T11:20:25.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5t+pmqvd+CRAAFjRgg29SiVKugouHwdaK8Q=="}' 2024-09-18T11:20:25.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d856b86-98bc-4571-9567-b228ca0d10fc -i td/test-erasure-eio/5/new.json 2024-09-18T11:20:26.085 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:20:26.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-09-18T11:20:26.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5t+pmqvd+CRAAFjRgg29SiVKugouHwdaK8Q== --osd-uuid 5d856b86-98bc-4571-9567-b228ca0d10fc 2024-09-18T11:20:26.124 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.120+0000 7f69ad487540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:26.268 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.236+0000 7f69ad487540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:26.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.268+0000 7f69ad487540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:26.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.268+0000 7f69ad487540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:20:26.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.268+0000 7f69ad487540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:20:26.272 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.268+0000 7f69ad487540 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-09-18T11:20:26.563 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:26.556+0000 7f69ad487540 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-09-18T11:20:33.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-09-18T11:20:33.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:20:33.102 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd5 key to auth repository 2024-09-18T11:20:33.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-18T11:20:33.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:20:33.645 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.5 2024-09-18T11:20:33.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-18T11:20:33.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:20:33.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:20:33.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:20:33.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:20:33.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:33.672+0000 7f02dc161540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:33.678 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:33.672+0000 7f02dc161540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:33.680 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:33.676+0000 7f02dc161540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:20:34.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-18T11:20:34.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:20:34.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-18T11:20:34.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:20:34.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:20:34.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:34.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:20:34.098 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:20:34.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:34.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:34.237 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:34.232+0000 7f02dc161540 -1 Falling back to public interface 2024-09-18T11:20:34.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:35.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:35.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:35.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:20:35.535 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:20:35.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:35.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:35.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:36.986 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:20:36.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:20:36.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:36.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:37.008 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:20:37.004+0000 7f02dc161540 -1 osd.5 0 log_to_monitors true 2024-09-18T11:20:37.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:38.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:38.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:38.549 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:20:38.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:20:38.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:38.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:39.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:40.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:40.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:40.249 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:20:40.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:20:40.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:40.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:41.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:42.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:42.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:42.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:20:42.030 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:20:42.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:42.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:42.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:43.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:43.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:43.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:20:43.582 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:20:43.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:43.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:44.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:45.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:45.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:45.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:20:45.003 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:20:45.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:45.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:45.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:46.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:46.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:46.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:20:46.495 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:20:46.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:46.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:47.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:48.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:48.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:48.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:20:48.051 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:20:48.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:48.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:48.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:49.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:49.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:49.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:20:49.572 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:20:49.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:49.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:50.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:51.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:51.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:51.014 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:20:51.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:20:51.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:51.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:51.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:52.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:52.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:52.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:20:52.498 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:20:52.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:52.498 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:53.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:54.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:54.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:54.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:20:54.237 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:20:54.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:54.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:55.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:56.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:56.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:56.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:20:56.359 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:20:56.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:56.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:56.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:57.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:57.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:57.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:20:57.901 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:20:57.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:57.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:20:58.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:20:59.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:20:59.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:20:59.318 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:20:59.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:20:59.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:20:59.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:21:00.246 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:01.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:01.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:01.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:21:01.249 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:21:01.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:01.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:21:02.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:03.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:03.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:03.191 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:21:03.191 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:21:03.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:03.192 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-18T11:21:03.721 INFO:tasks.workunit.client.0.smithi174.stdout:osd.5 up in weight 1 up_from 51 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/191916,v1:127.0.0.1:6843/191916] [v2:127.0.0.1:6844/191916,v1:127.0.0.1:6845/191916] exists,up 5d856b86-98bc-4571-9567-b228ca0d10fc 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:21:03.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:21:03.724 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:21:03.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:21:03.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:21:03.726 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:21:03.726 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:21:03.726 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:21:03.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-09-18T11:21:03.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:21:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2683906-672a-42aa-aaab-0b1484bf8d1a 2024-09-18T11:21:03.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 f2683906-672a-42aa-aaab-0b1484bf8d1a' 2024-09-18T11:21:03.729 INFO:tasks.workunit.client.0.smithi174.stdout:add osd6 f2683906-672a-42aa-aaab-0b1484bf8d1a 2024-09-18T11:21:03.729 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:21:03.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCft+pmGG5oLBAAEqtogOIkS7UmuiGLX12NxA== 2024-09-18T11:21:03.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCft+pmGG5oLBAAEqtogOIkS7UmuiGLX12NxA=="}' 2024-09-18T11:21:03.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2683906-672a-42aa-aaab-0b1484bf8d1a -i td/test-erasure-eio/6/new.json 2024-09-18T11:21:04.570 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:21:04.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-09-18T11:21:04.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCft+pmGG5oLBAAEqtogOIkS7UmuiGLX12NxA== --osd-uuid f2683906-672a-42aa-aaab-0b1484bf8d1a 2024-09-18T11:21:04.619 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.612+0000 7f7f36215540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:04.724 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.720+0000 7f7f36215540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:04.726 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.720+0000 7f7f36215540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:04.727 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.724+0000 7f7f36215540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:21:04.727 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.724+0000 7f7f36215540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:21:04.728 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.724+0000 7f7f36215540 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-09-18T11:21:04.860 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:04.856+0000 7f7f36215540 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-09-18T11:21:11.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-09-18T11:21:11.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:21:11.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-09-18T11:21:11.174 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd6 key to auth repository 2024-09-18T11:21:11.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:21:11.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-09-18T11:21:11.754 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.6 2024-09-18T11:21:11.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:21:11.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:21:11.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:21:11.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:21:11.783 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:11.780+0000 7f0d2c690540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:11.786 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:11.780+0000 7f0d2c690540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:11.787 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:11.784+0000 7f0d2c690540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:21:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-09-18T11:21:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:21:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-09-18T11:21:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:21:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:21:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:21:12.132 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:21:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:12.333 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:12.328+0000 7f0d2c690540 -1 Falling back to public interface 2024-09-18T11:21:12.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:13.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:13.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:13.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:21:13.565 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:21:13.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:13.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:14.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:14.857 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:21:14.852+0000 7f0d2c690540 -1 osd.6 0 log_to_monitors true 2024-09-18T11:21:15.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:15.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:15.009 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:21:15.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:21:15.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:15.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:15.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:16.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:16.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:16.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:21:16.617 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:21:16.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:16.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:17.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:18.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:18.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:18.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:21:18.353 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:21:18.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:18.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:18.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:19.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:19.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:19.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:21:19.971 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:21:19.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:19.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:20.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:21.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:21.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:21.510 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:21:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:21:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:21.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:22.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:23.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:23.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:23.035 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:21:23.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:21:23.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:23.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:23.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:24.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:24.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:24.522 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:21:24.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:21:24.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:24.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:25.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:26.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:26.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:26.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:21:26.064 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:21:26.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:26.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:26.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:27.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:27.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:27.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:21:27.587 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:21:27.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:27.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:28.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:29.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:29.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:29.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:21:29.119 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:21:29.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:29.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:29.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:30.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:30.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:30.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:21:30.589 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:21:30.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:30.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:31.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:32.073 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:21:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:21:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:32.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:33.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:33.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:33.572 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:21:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:21:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:33.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:34.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:35.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:35.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:35.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:21:35.593 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:21:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:35.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:36.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:37.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:37.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:37.415 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:21:37.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:21:37.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:37.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:38.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:39.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:39.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:39.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:21:39.175 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:21:39.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:39.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:39.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:21:40.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:21:40.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:21:40.886 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:21:40.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:21:40.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:21:40.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-09-18T11:21:41.402 INFO:tasks.workunit.client.0.smithi174.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/193717,v1:127.0.0.1:6851/193717] [v2:127.0.0.1:6852/193717,v1:127.0.0.1:6853/193717] exists,up f2683906-672a-42aa-aaab-0b1484bf8d1a 2024-09-18T11:21:41.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:21:41.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:21:41.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:21:41.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:21:41.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:21:41.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:21:41.404 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:21:41.404 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:21:41.404 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:21:41.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:21:41.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:21:41.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:21:41.489 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:21:41.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:21:41.501 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:17:29.247+0000 7fb536964540 0 load: jerasure load: lrc 2024-09-18T11:21:41.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:21:41.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-09-18T11:21:42.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:21:42.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:21:44.050 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:21:44.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:21:45.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:21:45.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:21:45.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:21:45.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:21:45.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:21:45.064 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:21:45.066 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:21:45.066 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:21:45.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:21:45.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:21:45.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:21:45.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:21:45.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:21:45.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:21:45.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:21:45.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:21:45.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:21:45.745 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:21:45.745 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:21:45.745 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:21:45.745 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:21:45.745 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:21:45.746 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:21:45.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:21:45.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:45.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:21:45.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803832 2024-09-18T11:21:45.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803832 2024-09-18T11:21:45.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832' 2024-09-18T11:21:45.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:45.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:21:46.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574895 2024-09-18T11:21:46.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574895 2024-09-18T11:21:46.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-55834574895' 2024-09-18T11:21:46.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:46.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:21:46.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378671 2024-09-18T11:21:46.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378671 2024-09-18T11:21:46.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-55834574895 2-81604378671' 2024-09-18T11:21:46.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:46.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:21:46.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051620 2024-09-18T11:21:46.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051620 2024-09-18T11:21:46.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-55834574895 2-81604378671 3-124554051620' 2024-09-18T11:21:46.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:46.360 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:21:46.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659163 2024-09-18T11:21:46.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659163 2024-09-18T11:21:46.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-55834574895 2-81604378671 3-124554051620 4-176093659163' 2024-09-18T11:21:46.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:46.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:21:46.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332122 2024-09-18T11:21:46.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332122 2024-09-18T11:21:46.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-55834574895 2-81604378671 3-124554051620 4-176093659163 5-219043332122' 2024-09-18T11:21:46.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:46.659 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:21:46.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=261993005072 2024-09-18T11:21:46.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 261993005072 2024-09-18T11:21:46.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-55834574895 2-81604378671 3-124554051620 4-176093659163 5-219043332122 6-261993005072' 2024-09-18T11:21:46.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:46.812 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803832 2024-09-18T11:21:46.813 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:46.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:21:46.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803832 2024-09-18T11:21:46.816 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:46.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803832 2024-09-18T11:21:46.818 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803832 2024-09-18T11:21:46.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-18T11:21:46.819 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:21:47.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803829 -lt 25769803832 2024-09-18T11:21:47.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:21:48.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:21:48.219 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:21:48.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803832 -lt 25769803832 2024-09-18T11:21:48.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:48.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574895 2024-09-18T11:21:48.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:48.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:21:48.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574895 2024-09-18T11:21:48.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:48.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574895 2024-09-18T11:21:48.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574895' 2024-09-18T11:21:48.683 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574895 2024-09-18T11:21:48.683 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:21:49.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574895 -lt 55834574895 2024-09-18T11:21:49.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:49.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378671 2024-09-18T11:21:49.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:49.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:21:49.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378671 2024-09-18T11:21:49.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378671 2024-09-18T11:21:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378671' 2024-09-18T11:21:49.176 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378671 2024-09-18T11:21:49.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:21:49.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378672 -lt 81604378671 2024-09-18T11:21:49.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:49.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051620 2024-09-18T11:21:49.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:49.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:21:49.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051620 2024-09-18T11:21:49.606 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:49.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051620 2024-09-18T11:21:49.608 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051620 2024-09-18T11:21:49.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051620' 2024-09-18T11:21:49.609 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:21:50.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051620 -lt 124554051620 2024-09-18T11:21:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:50.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-176093659163 2024-09-18T11:21:50.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:50.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:21:50.042 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-176093659163 2024-09-18T11:21:50.042 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:50.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659163 2024-09-18T11:21:50.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 176093659163' 2024-09-18T11:21:50.044 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 176093659163 2024-09-18T11:21:50.045 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:21:50.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659163 -lt 176093659163 2024-09-18T11:21:50.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:50.555 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-219043332122 2024-09-18T11:21:50.555 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:50.557 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:21:50.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-219043332122 2024-09-18T11:21:50.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:50.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332122 2024-09-18T11:21:50.560 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 219043332122 2024-09-18T11:21:50.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 219043332122' 2024-09-18T11:21:50.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:21:50.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332123 -lt 219043332122 2024-09-18T11:21:50.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:50.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:50.988 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 6-261993005072 2024-09-18T11:21:50.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=6 2024-09-18T11:21:50.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 6-261993005072 2024-09-18T11:21:50.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:50.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=261993005072 2024-09-18T11:21:50.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.6 seq 261993005072' 2024-09-18T11:21:50.992 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.6 seq 261993005072 2024-09-18T11:21:50.992 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-18T11:21:51.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 261993005072 -lt 261993005072 2024-09-18T11:21:51.397 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:21:51.397 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:21:51.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:21:51.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:21:51.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:21:51.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:21:51.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:21:51.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:21:51.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:21:51.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:21:51.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:21:52.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:21:52.357 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:21:52.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:21:52.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:21:52.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:21:52.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:21:52.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:21:52.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:21:52.880 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=myobject 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:21:53.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure myobject 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:21:53.047 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:21:53.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:21:53.559 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:21:53.559 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:21:53.559 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:21:53.559 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:21:53.559 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-18T11:21:53.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:21:53.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T11:21:53.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:21:53.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T11:21:53.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:21:53.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T11:21:53.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:21:53.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T11:21:53.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:21:53.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:21:53.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:21:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T11:21:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:21:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:21:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:21:53.565 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:21:53.566 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:21:53.566 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:21:53.566 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:21:53.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:21:53.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:21:53.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:21:53.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T11:21:53.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:21:53.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:21:53.658 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:21:53.658 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:21:53.658 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:21:53.658 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:21:53.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:21:53.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-09-18T11:21:53.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:21:53.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-09-18T11:21:53.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:21:53.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-18T11:21:53.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-09-18T11:21:53.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:21:54.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-09-18T11:21:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-09-18T11:21:54.196 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:21:54.196 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:21:54.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:21:54.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:21:54.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:21:54.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:21:54.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-09-18T11:21:55.274 INFO:tasks.workunit.client.0.smithi174.stderr:osd.6 is already down. 2024-09-18T11:21:55.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-09-18T11:21:56.134 INFO:tasks.workunit.client.0.smithi174.stderr:osd.6 is already out. 2024-09-18T11:21:56.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-09-18T11:21:56.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:21:56.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:21:56.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:21:56.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:21:56.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:21:56.144 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:21:56.144 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:21:56.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:21:56.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:21:56.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:21:56.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:21:56.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:21:56.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:21:56.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:21:56.349 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:4 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:5 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:6' 2024-09-18T11:21:56.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:21:56.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:56.907 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:21:57.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803836 2024-09-18T11:21:57.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803836 2024-09-18T11:21:57.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836' 2024-09-18T11:21:57.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:57.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:21:57.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574899 2024-09-18T11:21:57.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574899 2024-09-18T11:21:57.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-55834574899' 2024-09-18T11:21:57.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:57.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:21:57.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378675 2024-09-18T11:21:57.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378675 2024-09-18T11:21:57.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-55834574899 2-81604378675' 2024-09-18T11:21:57.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:57.445 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:21:57.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051624 2024-09-18T11:21:57.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051624 2024-09-18T11:21:57.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-55834574899 2-81604378675 3-124554051624' 2024-09-18T11:21:57.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:57.598 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-18T11:21:57.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=176093659167 2024-09-18T11:21:57.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 176093659167 2024-09-18T11:21:57.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-55834574899 2-81604378675 3-124554051624 4-176093659167' 2024-09-18T11:21:57.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:57.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-18T11:21:57.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=219043332127 2024-09-18T11:21:57.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 219043332127 2024-09-18T11:21:57.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803836 1-55834574899 2-81604378675 3-124554051624 4-176093659167 5-219043332127' 2024-09-18T11:21:57.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:21:57.942 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-18T11:21:58.040 INFO:tasks.workunit.client.0.smithi174.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-18T11:21:58.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-18T11:21:58.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-18T11:21:58.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-18T11:21:58.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:58.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803836 2024-09-18T11:21:58.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:58.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:21:58.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803836 2024-09-18T11:21:58.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:58.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803836 2024-09-18T11:21:58.049 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803836 2024-09-18T11:21:58.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803836' 2024-09-18T11:21:58.049 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:21:58.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803836 -lt 25769803836 2024-09-18T11:21:58.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:58.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574899 2024-09-18T11:21:58.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:58.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:21:58.474 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574899 2024-09-18T11:21:58.474 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:58.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574899 2024-09-18T11:21:58.476 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574899 2024-09-18T11:21:58.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574899' 2024-09-18T11:21:58.477 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:21:58.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574899 -lt 55834574899 2024-09-18T11:21:58.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:58.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378675 2024-09-18T11:21:58.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:58.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:21:58.920 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378675 2024-09-18T11:21:58.920 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:58.922 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378675 2024-09-18T11:21:58.922 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378675 2024-09-18T11:21:58.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378675' 2024-09-18T11:21:58.923 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:21:59.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378676 -lt 81604378675 2024-09-18T11:21:59.412 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:59.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051624 2024-09-18T11:21:59.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:59.416 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:21:59.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051624 2024-09-18T11:21:59.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:59.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051624 2024-09-18T11:21:59.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051624' 2024-09-18T11:21:59.420 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051624 2024-09-18T11:21:59.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:21:59.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051624 -lt 124554051624 2024-09-18T11:21:59.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:21:59.854 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-176093659167 2024-09-18T11:21:59.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:21:59.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-18T11:21:59.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:21:59.858 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-176093659167 2024-09-18T11:21:59.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=176093659167 2024-09-18T11:21:59.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 176093659167' 2024-09-18T11:21:59.859 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.4 seq 176093659167 2024-09-18T11:21:59.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-18T11:22:00.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 176093659167 -lt 176093659167 2024-09-18T11:22:00.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:22:00.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-219043332127 2024-09-18T11:22:00.268 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:22:00.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-18T11:22:00.270 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-219043332127 2024-09-18T11:22:00.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:22:00.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=219043332127 2024-09-18T11:22:00.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 219043332127' 2024-09-18T11:22:00.273 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.5 seq 219043332127 2024-09-18T11:22:00.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-18T11:22:00.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 219043332127 -lt 219043332127 2024-09-18T11:22:00.712 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:22:00.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:22:00.713 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:22:01.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:22:01.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:22:01.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:22:01.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:22:01.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:22:01.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:22:01.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:22:01.246 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:22:01.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:22:01.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:22:01.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:22:01.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:22:02.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:22:02.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:22:02.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:22:02.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-09-18T11:22:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:22:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:22:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-09-18T11:22:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:22:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:22:02.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-09-18T11:22:02.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:22:02.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:22:02.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:22:02.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:22:02.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:22:02.738 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:22:02.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:22:03.639 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:22:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:22:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:22:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:22:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:22:03.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:22:03.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:22:03.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:22:03.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:22:03.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:22:03.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:22:03.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:22:03.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:22:03.775 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:22:03.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:22:03.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:22:03.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:22:03.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:22:03.779 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:22:03.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:22:03.780 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:22:03.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:22:03.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:22:03.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:22:03.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:22:03.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:03.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:03.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:22:03.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:22:03.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:22:03.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:22:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:22:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:22:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:22:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:22:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:22:03.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:22:03.824 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:22:03.824 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:22:03.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:22:03.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:22:03.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:22:03.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:22:03.829 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:22:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:22:03.831 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:22:03.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:22:03.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:22:03.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:22:03.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:22:03.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:22:03.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:22:03.836 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:22:03.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:22:03.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:22:03.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:22:03.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:22:03.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:03.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:03.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:22:03.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:22:03.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:22:03.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:22:03.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:22:03.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:03.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:03.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:22:03.849 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:22:03.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:22:03.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:22:03.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:22:04.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:22:04.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:22:04.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:22:04.238 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:22:04.238 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:04.238 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:04.238 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:22:04.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:22:04.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:22:04.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:22:04.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:22:04.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:22:04.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:22:04.507 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:22:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:22:04.507 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:22:04.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:22:04.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:22:04.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:22:04.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:22:04.600 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:22:04.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:22:04.601 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:22:04.601 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:22:04.601 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:22:04.602 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:04.602 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:04.602 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:04.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:22:04.603 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:22:04.603 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:22:04.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:22:04.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:22:04.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:22:04.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:22:04.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:22:04.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:22:04.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:22:05.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:22:05.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:22:05.601 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:22:05.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:22:05.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:05.602 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:05.602 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:22:05.603 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:22:05.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:22:05.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:22:05.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:22:05.959 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:22:05.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:22:06.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:22:06.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:22:06.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:22:06.978 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:22:06.978 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:06.978 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:06.978 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:22:06.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:22:06.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:22:07.057 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:22:07.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:22:07.069 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:22:04.275+0000 7f6247989a00 0 load: jerasure load: lrc 2024-09-18T11:22:07.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-09-18T11:22:07.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-09-18T11:22:07.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-09-18T11:22:07.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:22:07.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:22:07.071 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:22:07.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:22:07.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:22:07.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:22:07.079 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:22:07.079 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:22:07.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:22:07.081 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:22:07.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:22:07.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7499b80a-96a1-4550-aa2c-373191e4a661 2024-09-18T11:22:07.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7499b80a-96a1-4550-aa2c-373191e4a661' 2024-09-18T11:22:07.082 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 7499b80a-96a1-4550-aa2c-373191e4a661 2024-09-18T11:22:07.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:22:07.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDft+pmks8DBhAAzhyXxqSGo+H15MgVCOKOXA== 2024-09-18T11:22:07.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDft+pmks8DBhAAzhyXxqSGo+H15MgVCOKOXA=="}' 2024-09-18T11:22:07.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7499b80a-96a1-4550-aa2c-373191e4a661 -i td/test-erasure-eio/0/new.json 2024-09-18T11:22:07.452 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:22:07.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:22:07.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDft+pmks8DBhAAzhyXxqSGo+H15MgVCOKOXA== --osd-uuid 7499b80a-96a1-4550-aa2c-373191e4a661 2024-09-18T11:22:07.499 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.495+0000 7fca35443540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:07.502 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.495+0000 7fca35443540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:07.504 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.499+0000 7fca35443540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:07.505 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.499+0000 7fca35443540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:22:07.505 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.499+0000 7fca35443540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:22:07.505 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.499+0000 7fca35443540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:22:07.593 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:07.587+0000 7fca35443540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:22:13.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:22:13.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:22:13.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:22:13.302 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:22:13.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:22:13.841 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:22:13.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:22:13.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:22:13.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:22:13.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:22:13.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:22:13.872 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:13.867+0000 7f88371bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:13.875 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:13.871+0000 7f88371bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:13.876 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:13.871+0000 7f88371bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:22:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:22:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:22:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:22:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:22:14.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:14.281 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:22:14.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:22:14.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:14.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:14.417 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:14.411+0000 7f88371bd540 -1 Falling back to public interface 2024-09-18T11:22:14.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:15.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:15.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:15.686 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:22:15.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:22:15.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:15.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:16.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:16.962 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:16.955+0000 7f88371bd540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:22:17.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:17.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:17.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:22:17.106 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:22:17.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:17.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:17.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:18.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:18.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:18.620 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:22:18.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:22:18.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:18.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:19.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:22:20.036 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:22:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:20.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:21.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:22:21.431 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:22:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:21.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:21.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:22.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:22.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:22.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:22:22.912 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:22:22.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:22.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:23.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:24.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:24.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:24.366 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:22:24.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:22:24.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:24.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:24.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:25.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:25.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:25.855 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:22:25.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:22:25.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:25.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:26.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:27.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:27.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:27.313 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:22:27.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:22:27.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:27.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:27.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:28.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:28.807 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:22:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:22:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:28.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:29.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:30.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:30.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:30.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:22:30.513 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:22:30.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:30.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:31.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:32.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:32.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:32.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:22:32.163 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:22:32.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:32.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:33.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:34.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:34.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:34.343 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:22:34.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:22:34.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:34.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:35.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:36.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:36.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:36.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:22:36.208 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:22:36.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:36.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:37.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:38.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:38.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:38.320 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:22:38.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:22:38.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:38.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:39.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:40.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:40.111 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:40.112 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:22:40.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:22:40.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:40.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:40.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:41.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:41.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:41.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:22:41.671 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:22:41.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:41.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:22:42.285 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/197308,v1:127.0.0.1:6803/197308] [v2:127.0.0.1:6804/197308,v1:127.0.0.1:6805/197308] exists,up 7499b80a-96a1-4550-aa2c-373191e4a661 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:22:42.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:22:42.288 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:22:42.289 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:22:42.289 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:22:42.290 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:22:42.290 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:22:42.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:22:42.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:22:42.292 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:22:42.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:22:42.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c82771f6-0a9f-4710-9729-929630717b79' 2024-09-18T11:22:42.294 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:22:42.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:22:42.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACuOpmm/6dEhAAT9pI32OxdaWu/7+9DBYClg== 2024-09-18T11:22:42.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACuOpmm/6dEhAAT9pI32OxdaWu/7+9DBYClg=="}' 2024-09-18T11:22:42.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c82771f6-0a9f-4710-9729-929630717b79 -i td/test-erasure-eio/1/new.json 2024-09-18T11:22:42.856 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:22:42.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:22:42.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACuOpmm/6dEhAAT9pI32OxdaWu/7+9DBYClg== --osd-uuid c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:22:42.904 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:42.898+0000 7f3a5b677540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:42.908 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:42.902+0000 7f3a5b677540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:42.909 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:42.902+0000 7f3a5b677540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:42.911 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:42.906+0000 7f3a5b677540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:22:42.911 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:42.906+0000 7f3a5b677540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:22:42.911 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:42.906+0000 7f3a5b677540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:22:43.011 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:43.006+0000 7f3a5b677540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:22:49.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:22:49.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:22:49.050 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:22:49.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:22:49.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:22:49.587 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:22:49.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:22:49.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:22:49.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:22:49.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:22:49.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:22:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:49.610+0000 7fb2e49fe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:49.619 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:49.614+0000 7fb2e49fe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:49.620 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:49.614+0000 7fb2e49fe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:50.038 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:22:50.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:22:50.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:50.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:50.202+0000 7fb2e49fe540 -1 Falling back to public interface 2024-09-18T11:22:50.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:51.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:51.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:51.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:22:51.480 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:22:51.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:51.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:51.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:52.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:52.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:52.983 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:22:52.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:22:52.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:52.984 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:53.012 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:22:53.006+0000 7fb2e49fe540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:22:53.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:54.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:54.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:54.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:22:54.521 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:22:54.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:54.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:54.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:22:55.976 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:22:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:55.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:56.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:57.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:57.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:57.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:22:57.401 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:22:57.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:57.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:57.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:22:58.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:22:58.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:22:58.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:22:58.988 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:22:58.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:22:58.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:22:59.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:00.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:00.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:00.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:23:00.399 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:23:00.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:00.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:00.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:01.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:01.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:01.834 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:23:01.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:23:01.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:01.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:02.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:03.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:03.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:03.297 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:23:03.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:23:03.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:03.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:03.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:04.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:04.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:04.847 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:23:04.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:23:04.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:04.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:05.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:06.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:06.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:06.503 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:23:06.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:23:06.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:06.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:07.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:08.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:08.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:08.033 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:23:08.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:23:08.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:08.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:09.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:10.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:23:10.107 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:23:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:10.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:10.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:11.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:11.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:11.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:23:11.890 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:23:11.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:11.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:12.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:13.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:13.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:13.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:23:13.702 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:23:13.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:13.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:14.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:15.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:15.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:15.704 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:23:15.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:23:15.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:15.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:16.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:17.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:17.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:17.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:23:17.303 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:23:17.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:17.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:23:17.849 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/199083,v1:127.0.0.1:6811/199083] [v2:127.0.0.1:6812/199083,v1:127.0.0.1:6813/199083] exists,up c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:23:17.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:23:17.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:23:17.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:23:17.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:23:17.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:23:17.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:23:17.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:23:17.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:23:17.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:23:17.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:23:17.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:23:17.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e23d5821-f65f-4b88-b8cc-04764f6aa285 2024-09-18T11:23:17.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e23d5821-f65f-4b88-b8cc-04764f6aa285' 2024-09-18T11:23:17.858 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 e23d5821-f65f-4b88-b8cc-04764f6aa285 2024-09-18T11:23:17.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:23:17.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAluOpmFNNVNBAAfuIgGV+1V56Zup3awFJjmw== 2024-09-18T11:23:17.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAluOpmFNNVNBAAfuIgGV+1V56Zup3awFJjmw=="}' 2024-09-18T11:23:17.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e23d5821-f65f-4b88-b8cc-04764f6aa285 -i td/test-erasure-eio/2/new.json 2024-09-18T11:23:18.455 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:23:18.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:23:18.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAluOpmFNNVNBAAfuIgGV+1V56Zup3awFJjmw== --osd-uuid e23d5821-f65f-4b88-b8cc-04764f6aa285 2024-09-18T11:23:18.502 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.494+0000 7ff75e8a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:18.562 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.554+0000 7ff75e8a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:18.564 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.558+0000 7ff75e8a5540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:18.565 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.558+0000 7ff75e8a5540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:23:18.565 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.558+0000 7ff75e8a5540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:23:18.565 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.558+0000 7ff75e8a5540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:23:18.639 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:18.634+0000 7ff75e8a5540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:23:24.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:23:24.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:23:24.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:23:24.904 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:23:24.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:23:25.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:23:25.451 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:23:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:23:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:23:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:23:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:23:25.480 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:25.474+0000 7f6da0bca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:25.483 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:25.478+0000 7f6da0bca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:25.484 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:25.478+0000 7f6da0bca540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:25.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:23:25.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:23:25.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:23:25.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:23:25.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:23:25.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:25.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:23:25.892 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:23:25.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:25.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:26.069 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:26.062+0000 7f6da0bca540 -1 Falling back to public interface 2024-09-18T11:23:26.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:27.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:27.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:27.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:23:27.285 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:23:27.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:27.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:27.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:28.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:28.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:28.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:23:28.696 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:23:28.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:28.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:28.812 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:28.806+0000 7f6da0bca540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:23:29.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:30.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:30.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:30.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:23:30.153 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:23:30.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:30.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:30.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:31.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:31.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:31.639 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:23:31.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:23:31.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:31.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:32.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:33.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:33.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:33.184 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:23:33.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:23:33.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:33.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:33.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:34.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:34.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:34.754 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:23:34.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:23:34.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:34.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:35.295 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:36.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:36.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:36.297 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:23:36.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:23:36.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:36.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:36.836 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:37.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:37.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:37.839 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:23:37.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:23:37.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:37.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:38.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:39.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:39.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:39.301 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:23:39.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:23:39.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:39.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:39.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:40.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:40.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:40.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:23:40.852 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:23:40.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:40.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:41.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:42.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:42.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:42.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:23:42.259 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:23:42.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:42.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:42.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:43.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:43.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:43.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:23:43.707 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:23:43.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:43.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:44.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:45.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:45.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:45.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:23:45.441 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:23:45.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:45.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:46.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:47.086 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:23:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:23:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:47.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:48.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:48.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:48.620 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:23:48.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:23:48.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:48.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:49.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:50.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:50.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:50.593 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:23:50.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:23:50.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:50.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:51.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:52.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:52.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:52.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:23:52.418 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:23:52.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:52.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:53.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:23:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:23:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:23:54.118 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:23:54.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:23:54.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:23:54.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:23:54.786 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/200858,v1:127.0.0.1:6819/200858] [v2:127.0.0.1:6820/200858,v1:127.0.0.1:6821/200858] exists,up e23d5821-f65f-4b88-b8cc-04764f6aa285 2024-09-18T11:23:54.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:23:54.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:23:54.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:23:54.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:23:54.794 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:23:54.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:23:54.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 088c3fab-467e-44cc-bdeb-542cb0a1ca7b' 2024-09-18T11:23:54.794 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:23:54.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:23:54.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKuOpmK8uvMBAAjwtSq2OG4vqM2Paj45nceQ== 2024-09-18T11:23:54.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKuOpmK8uvMBAAjwtSq2OG4vqM2Paj45nceQ=="}' 2024-09-18T11:23:54.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 088c3fab-467e-44cc-bdeb-542cb0a1ca7b -i td/test-erasure-eio/3/new.json 2024-09-18T11:23:55.519 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:23:55.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:23:55.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKuOpmK8uvMBAAjwtSq2OG4vqM2Paj45nceQ== --osd-uuid 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:23:55.571 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.565+0000 7ff413c1b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:55.575 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.569+0000 7ff413c1b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:55.577 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.569+0000 7ff413c1b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:23:55.578 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.569+0000 7ff413c1b540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:23:55.578 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.569+0000 7ff413c1b540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:23:55.578 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.569+0000 7ff413c1b540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:23:55.667 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:23:55.661+0000 7ff413c1b540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:24:02.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:24:02.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:24:02.217 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:24:02.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:24:02.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:24:02.751 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:24:02.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:24:02.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:24:02.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:24:02.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:24:02.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:24:02.781 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:02.773+0000 7fb2daa78540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:02.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:02.777+0000 7fb2daa78540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:02.785 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:02.777+0000 7fb2daa78540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:24:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:24:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:24:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:24:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:24:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:03.201 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:24:03.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:24:03.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:03.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:03.325 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:03.317+0000 7fb2daa78540 -1 Falling back to public interface 2024-09-18T11:24:03.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:04.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:04.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:04.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:24:04.573 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:24:04.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:04.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:05.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:06.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:06.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:06.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:24:06.056 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:24:06.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:06.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:06.111 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:06.105+0000 7fb2daa78540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:24:06.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:07.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:07.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:07.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:24:07.471 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:24:07.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:07.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:08.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:09.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:09.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:09.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:24:09.215 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:24:09.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:09.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:09.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:10.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:10.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:10.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:24:10.849 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:24:10.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:10.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:11.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:12.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:12.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:12.409 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:24:12.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:24:12.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:12.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:12.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:13.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:13.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:13.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:24:13.955 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:24:13.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:13.955 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:14.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:15.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:15.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:15.396 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:24:15.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:24:15.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:15.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:15.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:16.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:16.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:16.972 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:24:16.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:24:16.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:16.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:17.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:18.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:18.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:18.431 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:24:18.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:24:18.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:18.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:18.977 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:19.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:19.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:19.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:24:19.979 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:24:19.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:19.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:20.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:21.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:21.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:21.402 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:24:21.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:24:21.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:21.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:21.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:22.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:22.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:22.812 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:24:22.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:24:22.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:22.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:23.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:24.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:24.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:24.850 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:24:24.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:24:24.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:24.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:25.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:26.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:26.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:26.528 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:24:26.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:24:26.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:26.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:27.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:28.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:28.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:28.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:24:28.437 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:24:28.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:28.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:29.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:30.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:30.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:30.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:24:30.267 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:24:30.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:30.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:31.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:32.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:32.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:32.107 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:24:32.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:24:32.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:32.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:32.580 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/202659,v1:127.0.0.1:6827/202659] [v2:127.0.0.1:6828/202659,v1:127.0.0.1:6829/202659] exists,up 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:24:32.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:24:32.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:24:32.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:24:32.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:24:32.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:24:32.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:24:32.582 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:24:32.582 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:24:32.582 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:24:32.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:24:32.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:24:32.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:24:32.669 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:24:32.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:24:32.683 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:22:14.419+0000 7f88371bd540 0 load: jerasure load: lrc 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:24:32.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:24:33.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:24:33.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:24:34.533 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:24:34.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:24:35.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:24:35.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:24:35.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:24:35.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:24:35.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:24:35.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:24:35.546 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:24:35.546 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:24:35.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:24:35.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:24:35.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:24:35.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:24:35.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:24:35.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:24:35.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:24:35.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:24:36.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:24:36.341 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:24:36.341 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:24:36.341 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:24:36.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:24:36.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:36.342 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:24:36.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803811 2024-09-18T11:24:36.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803811 2024-09-18T11:24:36.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811' 2024-09-18T11:24:36.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:36.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:24:36.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574878 2024-09-18T11:24:36.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574878 2024-09-18T11:24:36.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574878' 2024-09-18T11:24:36.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:36.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:24:36.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378646 2024-09-18T11:24:36.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378646 2024-09-18T11:24:36.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574878 2-81604378646' 2024-09-18T11:24:36.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:36.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:24:37.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084298 2024-09-18T11:24:37.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084298 2024-09-18T11:24:37.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574878 2-81604378646 3-120259084298' 2024-09-18T11:24:37.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:37.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803811 2024-09-18T11:24:37.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:37.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:24:37.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803811 2024-09-18T11:24:37.024 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:37.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803811 2024-09-18T11:24:37.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-18T11:24:37.026 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803811 2024-09-18T11:24:37.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:24:37.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-18T11:24:37.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:37.475 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574878 2024-09-18T11:24:37.475 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:37.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:24:37.479 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574878 2024-09-18T11:24:37.479 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:37.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574878 2024-09-18T11:24:37.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-09-18T11:24:37.482 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574878 2024-09-18T11:24:37.482 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:24:37.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574878 -lt 55834574878 2024-09-18T11:24:37.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:37.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378646 2024-09-18T11:24:37.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:37.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:24:37.923 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378646 2024-09-18T11:24:37.923 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:37.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378646 2024-09-18T11:24:37.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-09-18T11:24:37.925 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378646 2024-09-18T11:24:37.925 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:24:38.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378644 -lt 81604378646 2024-09-18T11:24:38.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:24:39.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:24:39.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:24:39.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378646 -lt 81604378646 2024-09-18T11:24:39.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:39.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084298 2024-09-18T11:24:39.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:39.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:24:39.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084298 2024-09-18T11:24:39.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:39.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084298 2024-09-18T11:24:39.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084298' 2024-09-18T11:24:39.728 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084298 2024-09-18T11:24:39.728 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:24:40.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084299 -lt 120259084298 2024-09-18T11:24:40.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:24:40.144 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:24:40.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:24:40.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:24:40.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:24:40.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:24:40.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:24:40.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:24:40.680 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:24:40.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:24:40.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:24:41.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:24:41.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:24:41.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:24:41.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:24:41.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-18T11:24:41.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79554-0-10 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79554-0-10 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79554-0-10 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:24:41.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79554-0-10 td/test-erasure-eio/ORIGINAL 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79554-0-10 td/test-erasure-eio 0 10 set 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-0-10 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:24:41.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:24:41.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-0-10 2024-09-18T11:24:41.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:24:41.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-0-10 2024-09-18T11:24:41.769 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-0-10 2024-09-18T11:24:41.769 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:24:42.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:24:42.216 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:24:42.216 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:24:42.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:24:42.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:24:42.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-18T11:24:42.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:24:42.951 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:24:42.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:24:42.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-18T11:24:42.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:10 bytes copied, 0.000182937 s, 54.7 kB/s 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:24:42.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:24:42.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T11:24:42.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:24:42.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-18T11:24:42.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:24:42.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:24:42.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:24:42.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:24:42.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:24:44.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-18T11:24:44.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:24:46.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:24:46.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:24:46.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T11:24:46.262 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.3 2024-09-18T11:24:46.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:24:46.264 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-18T11:24:46.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T11:24:46.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:24:46.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:24:46.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:24:46.293 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:46.285+0000 7fe36b0bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:46.297 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:46.289+0000 7fe36b0bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:46.298 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:46.289+0000 7fe36b0bd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:24:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T11:24:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:24:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:24:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:24:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:24:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:46.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:24:46.671 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:24:46.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:46.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:46.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:46.857+0000 7fe36b0bd540 -1 Falling back to public interface 2024-09-18T11:24:47.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:48.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:48.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:48.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:24:48.116 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:24:48.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:48.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:48.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:49.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:49.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:49.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:24:49.514 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:24:49.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:49.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:49.698 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:24:49.689+0000 7fe36b0bd540 -1 osd.3 37 log_to_monitors true 2024-09-18T11:24:50.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:24:51.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:24:51.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:24:51.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:24:51.029 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:24:51.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:24:51.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:24:51.541 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 up in weight 1 up_from 42 up_thru 42 down_at 39 last_clean_interval [28,37) [v2:127.0.0.1:6826/204602,v1:127.0.0.1:6827/204602] [v2:127.0.0.1:6828/204602,v1:127.0.0.1:6829/204602] exists,up 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:24:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:24:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:24:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:24:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:24:51.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:24:51.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:24:51.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:24:51.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:24:51.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:24:51.545 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:24:51.545 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:24:51.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:24:51.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:24:51.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:24:51.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:24:51.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:24:51.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:24:51.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:24:51.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:24:52.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:24:52.169 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:24:52.169 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:24:52.169 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:24:52.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:24:52.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:52.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:24:52.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803816 2024-09-18T11:24:52.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803816 2024-09-18T11:24:52.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816' 2024-09-18T11:24:52.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:52.367 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:24:52.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574883 2024-09-18T11:24:52.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574883 2024-09-18T11:24:52.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574883' 2024-09-18T11:24:52.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:52.529 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:24:52.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378651 2024-09-18T11:24:52.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378651 2024-09-18T11:24:52.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574883 2-81604378651' 2024-09-18T11:24:52.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:24:52.684 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:24:52.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626435 2024-09-18T11:24:52.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626435 2024-09-18T11:24:52.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574883 2-81604378651 3-180388626435' 2024-09-18T11:24:52.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:52.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803816 2024-09-18T11:24:52.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:52.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:24:52.843 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803816 2024-09-18T11:24:52.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:52.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803816 2024-09-18T11:24:52.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-18T11:24:52.846 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803816 2024-09-18T11:24:52.846 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:24:53.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803816 -lt 25769803816 2024-09-18T11:24:53.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:53.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574883 2024-09-18T11:24:53.302 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:53.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:24:53.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:53.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574883 2024-09-18T11:24:53.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574883 2024-09-18T11:24:53.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574883' 2024-09-18T11:24:53.306 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 55834574883 2024-09-18T11:24:53.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:24:53.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574884 -lt 55834574883 2024-09-18T11:24:53.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:53.756 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:53.756 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378651 2024-09-18T11:24:53.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:24:53.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378651 2024-09-18T11:24:53.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:53.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378651 2024-09-18T11:24:53.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378651' 2024-09-18T11:24:53.762 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378651 2024-09-18T11:24:53.762 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:24:54.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378651 -lt 81604378651 2024-09-18T11:24:54.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:24:54.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-180388626435 2024-09-18T11:24:54.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:24:54.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:24:54.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-180388626435 2024-09-18T11:24:54.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:24:54.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626435 2024-09-18T11:24:54.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 180388626435' 2024-09-18T11:24:54.209 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 180388626435 2024-09-18T11:24:54.209 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:24:54.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626434 -lt 180388626435 2024-09-18T11:24:54.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:24:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:24:55.590 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:24:56.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626435 -lt 180388626435 2024-09-18T11:24:56.003 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:24:56.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:24:56.004 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:24:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:24:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:24:56.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:24:56.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:24:56.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:24:56.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:24:56.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:24:56.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:24:56.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:24:57.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:24:57.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:24:57.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:24:57.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:24:57.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:24:57.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:24:57.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:24:57.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:24:58.663 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:24:58.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-0-10 2024-09-18T11:24:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:24:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:24:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-0-10 2024-09-18T11:24:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:24:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:24:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79554-0-10 td/test-erasure-eio/COPY 2024-09-18T11:24:58.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:24:58.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:24:58.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-18T11:24:58.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-18T11:24:58.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79554-0-10 td/test-erasure-eio 1 10 set 2024-09-18T11:24:58.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-0-10 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:24:58.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-0-10 2024-09-18T11:24:58.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:24:58.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-0-10 2024-09-18T11:24:58.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-0-10 2024-09-18T11:24:58.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:24:59.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:24:59.305 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:24:59.305 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:24:59.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:24:59.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:24:59.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-18T11:24:59.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:25:00.166 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:25:00.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:25:00.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-18T11:25:00.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-18T11:25:00.181 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:10 bytes copied, 0.000193414 s, 51.7 kB/s 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:25:00.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:25:00.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:25:00.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:25:00.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:25:00.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:25:00.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:25:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:25:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:25:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:25:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:25:00.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:25:03.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:25:03.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:25:03.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:25:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:25:03.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:25:03.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:25:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:25:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:25:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:25:03.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:25:03.230 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:03.225+0000 7f0068306540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:03.233 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:03.229+0000 7f0068306540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:03.235 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:03.233+0000 7f0068306540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:25:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:25:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:25:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:25:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:25:03.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:03.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:25:03.614 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:25:03.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:03.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:03.797 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:03.793+0000 7f0068306540 -1 Falling back to public interface 2024-09-18T11:25:04.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:25:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:05.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:05.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:06.471 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:06.469+0000 7f0068306540 -1 osd.1 48 log_to_monitors true 2024-09-18T11:25:06.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:06.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:06.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:25:06.483 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:06.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:06.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:07.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:08.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:08.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:08.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:25:08.023 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:25:08.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:08.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [13,48) [v2:127.0.0.1:6810/206129,v1:127.0.0.1:6811/206129] [v2:127.0.0.1:6812/206129,v1:127.0.0.1:6813/206129] exists,up c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:25:08.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:25:08.503 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:25:08.503 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:25:08.503 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:25:08.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:25:08.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:25:08.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:25:08.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:25:08.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:25:08.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:25:08.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:25:08.723 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:09.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:25:09.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803821 2024-09-18T11:25:09.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803821 2024-09-18T11:25:09.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821' 2024-09-18T11:25:09.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:09.285 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:25:09.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=227633266691 2024-09-18T11:25:09.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 227633266691 2024-09-18T11:25:09.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-227633266691' 2024-09-18T11:25:09.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:09.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:25:09.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378657 2024-09-18T11:25:09.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378657 2024-09-18T11:25:09.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-227633266691 2-81604378657' 2024-09-18T11:25:09.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:09.586 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:25:09.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626440 2024-09-18T11:25:09.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626440 2024-09-18T11:25:09.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-227633266691 2-81604378657 3-180388626440' 2024-09-18T11:25:09.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:09.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803821 2024-09-18T11:25:09.735 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:09.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:25:09.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803821 2024-09-18T11:25:09.738 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803821 2024-09-18T11:25:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2024-09-18T11:25:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803821 2024-09-18T11:25:09.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:25:10.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803819 -lt 25769803821 2024-09-18T11:25:10.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:25:11.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:25:11.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:25:11.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803821 -lt 25769803821 2024-09-18T11:25:11.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:11.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-227633266691 2024-09-18T11:25:11.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:11.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:25:11.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-227633266691 2024-09-18T11:25:11.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:11.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=227633266691 2024-09-18T11:25:11.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 227633266691' 2024-09-18T11:25:11.678 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 227633266691 2024-09-18T11:25:11.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:25:12.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 227633266691 -lt 227633266691 2024-09-18T11:25:12.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:12.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378657 2024-09-18T11:25:12.167 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:12.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:25:12.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:12.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378657 2024-09-18T11:25:12.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378657 2024-09-18T11:25:12.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378657' 2024-09-18T11:25:12.171 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378657 2024-09-18T11:25:12.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:25:12.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378657 -lt 81604378657 2024-09-18T11:25:12.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:12.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:12.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-180388626440 2024-09-18T11:25:12.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:25:12.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-180388626440 2024-09-18T11:25:12.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:12.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626440 2024-09-18T11:25:12.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 180388626440' 2024-09-18T11:25:12.676 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 180388626440 2024-09-18T11:25:12.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:25:13.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626441 -lt 180388626440 2024-09-18T11:25:13.089 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:25:13.090 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:25:13.090 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:25:13.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:25:13.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:25:13.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:25:13.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:25:13.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:25:13.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:25:13.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:25:13.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:25:14.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:25:14.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:25:14.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:25:14.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:25:14.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:25:14.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:25:14.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:25:14.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:25:14.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:25:15.368 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-0-10 fail 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-0-10 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:25:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79554-0-10 td/test-erasure-eio/COPY 2024-09-18T11:25:15.537 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-size-79554-0-10: (5) Input/output error 2024-09-18T11:25:15.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:25:15.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:25:15.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-09-18T11:25:15.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-18T11:25:15.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79554-0-0 2024-09-18T11:25:15.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79554-0-0 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79554-0-0 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:25:15.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79554-0-0 td/test-erasure-eio/ORIGINAL 2024-09-18T11:25:15.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79554-0-0 td/test-erasure-eio 0 0 set 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-0-0 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:25:15.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:25:15.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-0-0 2024-09-18T11:25:15.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:25:15.701 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-0-0 2024-09-18T11:25:15.701 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-0-0 2024-09-18T11:25:15.702 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:25:16.113 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:25:16.113 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:16.113 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:25:16.113 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:25:16.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:25:16.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-18T11:25:16.114 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:25:16.882 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:25:16.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:25:16.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-18T11:25:16.895 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-18T11:25:16.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:25:16.897 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-18T11:25:16.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:25:16.898 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:25:16.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:25:16.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:25:16.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:25:18.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-18T11:25:18.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:20.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:25:20.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:25:20.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:25:20.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:25:20.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:25:20.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:25:20.199 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:25:20.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:25:20.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:25:20.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:25:20.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:25:20.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:25:20.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:25:20.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T11:25:20.203 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.3 2024-09-18T11:25:20.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:25:20.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-18T11:25:20.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T11:25:20.205 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:25:20.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:25:20.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:25:20.234 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:20.228+0000 7f46ef773540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:20.240 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:20.232+0000 7f46ef773540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:20.241 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:20.236+0000 7f46ef773540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:20.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T11:25:20.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:25:20.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:25:20.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:25:20.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:25:20.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:20.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:25:20.653 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:25:20.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:20.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:20.816 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:20.812+0000 7f46ef773540 -1 Falling back to public interface 2024-09-18T11:25:21.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:22.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:22.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:22.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:25:22.113 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:22.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:22.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:22.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:23.517 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:23.512+0000 7f46ef773540 -1 osd.3 58 log_to_monitors true 2024-09-18T11:25:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:25:23.550 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:23.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:23.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:24.119 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 up in weight 1 up_from 64 up_thru 53 down_at 60 last_clean_interval [42,58) [v2:127.0.0.1:6826/207673,v1:127.0.0.1:6827/207673] [v2:127.0.0.1:6828/207673,v1:127.0.0.1:6829/207673] exists,up 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:25:24.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:25:24.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:25:24.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:25:24.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:25:24.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:25:24.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:25:24.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:25:24.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:25:24.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:25:24.122 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:25:24.122 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:25:24.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:25:24.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:25:24.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:25:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:25:24.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:25:24.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:25:24.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:25:24.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:24.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:25:25.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803826 2024-09-18T11:25:25.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803826 2024-09-18T11:25:25.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826' 2024-09-18T11:25:25.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:25.157 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:25:25.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=227633266696 2024-09-18T11:25:25.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 227633266696 2024-09-18T11:25:25.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-227633266696' 2024-09-18T11:25:25.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:25.439 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:25:25.596 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378662 2024-09-18T11:25:25.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378662 2024-09-18T11:25:25.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-227633266696 2-81604378662' 2024-09-18T11:25:25.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:25.597 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:25:25.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906947 2024-09-18T11:25:25.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906947 2024-09-18T11:25:25.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803826 1-227633266696 2-81604378662 3-274877906947' 2024-09-18T11:25:25.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:25.784 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:25.784 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803826 2024-09-18T11:25:25.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:25:25.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803826 2024-09-18T11:25:25.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:25.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803826 2024-09-18T11:25:25.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-09-18T11:25:25.790 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803826 2024-09-18T11:25:25.790 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:25:26.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803824 -lt 25769803826 2024-09-18T11:25:26.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:25:27.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:25:27.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:25:27.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803827 -lt 25769803826 2024-09-18T11:25:27.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:27.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-227633266696 2024-09-18T11:25:27.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:27.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:25:27.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-227633266696 2024-09-18T11:25:27.669 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:27.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=227633266696 2024-09-18T11:25:27.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 227633266696' 2024-09-18T11:25:27.672 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 227633266696 2024-09-18T11:25:27.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:25:28.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 227633266696 -lt 227633266696 2024-09-18T11:25:28.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:28.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378662 2024-09-18T11:25:28.077 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:28.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:25:28.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378662 2024-09-18T11:25:28.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:28.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378662 2024-09-18T11:25:28.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378662' 2024-09-18T11:25:28.082 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378662 2024-09-18T11:25:28.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:25:28.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378662 -lt 81604378662 2024-09-18T11:25:28.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:28.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906947 2024-09-18T11:25:28.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:28.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:25:28.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906947 2024-09-18T11:25:28.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:28.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906947 2024-09-18T11:25:28.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906947' 2024-09-18T11:25:28.518 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 274877906947 2024-09-18T11:25:28.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:25:28.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906947 -lt 274877906947 2024-09-18T11:25:28.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:25:28.925 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:25:28.925 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:25:29.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:25:29.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:25:29.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:25:29.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:25:29.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:25:29.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:25:29.450 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:25:29.450 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:25:29.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:25:29.859 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:25:29.860 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:25:29.860 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:25:30.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:25:30.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:25:30.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:25:30.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:25:30.372 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:25:31.202 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:25:31.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-0-0 2024-09-18T11:25:31.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:25:31.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:25:31.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-0-0 2024-09-18T11:25:31.216 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:25:31.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:25:31.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79554-0-0 td/test-erasure-eio/COPY 2024-09-18T11:25:31.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:25:31.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:25:31.381 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79554-0-0 td/test-erasure-eio 1 0 set 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-0-0 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:25:31.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-18T11:25:31.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:25:31.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:25:31.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:25:31.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:25:31.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-0-0 2024-09-18T11:25:31.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:25:31.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-0-0 2024-09-18T11:25:31.386 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-0-0 2024-09-18T11:25:31.386 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:25:31.884 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:25:31.884 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-18T11:25:31.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:25:32.668 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:25:32.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:25:32.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-18T11:25:32.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-18T11:25:32.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:32.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:25:32.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:25:32.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:25:32.684 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:25:32.684 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:25:32.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:25:32.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:25:32.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:25:32.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:25:35.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:25:35.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:25:35.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:25:35.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:25:35.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:25:35.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:25:35.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:25:35.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:25:35.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:25:35.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:25:35.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:25:35.759 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:25:35.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:25:35.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:25:35.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:25:35.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:25:35.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:25:35.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:25:35.790 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:35.784+0000 7f4fbf777540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:35.824 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:35.820+0000 7f4fbf777540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:35.826 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:35.820+0000 7f4fbf777540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:25:36.226 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:25:36.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:36.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:36.376 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:36.372+0000 7f4fbf777540 -1 Falling back to public interface 2024-09-18T11:25:36.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:37.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:37.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:37.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:25:37.680 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:37.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:37.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:38.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:39.119 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:39.116+0000 7f4fbf777540 -1 osd.1 68 log_to_monitors true 2024-09-18T11:25:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:25:39.147 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:39.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:39.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:25:40.604 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:25:40.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:40.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:25:41.394 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 71 last_clean_interval [53,68) [v2:127.0.0.1:6810/209171,v1:127.0.0.1:6811/209171] [v2:127.0.0.1:6812/209171,v1:127.0.0.1:6813/209171] exists,up c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:25:41.395 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:25:41.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:25:41.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:25:41.397 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:25:41.397 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:25:41.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:25:41.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:25:41.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:25:41.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:25:41.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:25:41.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:25:41.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:25:41.611 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:25:42.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:25:42.027 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:42.027 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:42.027 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:25:42.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:25:42.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:42.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:25:42.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803832 2024-09-18T11:25:42.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803832 2024-09-18T11:25:42.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832' 2024-09-18T11:25:42.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:42.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:25:42.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=322122547203 2024-09-18T11:25:42.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 322122547203 2024-09-18T11:25:42.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-322122547203' 2024-09-18T11:25:42.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:42.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:25:42.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378667 2024-09-18T11:25:42.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378667 2024-09-18T11:25:42.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-322122547203 2-81604378667' 2024-09-18T11:25:42.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:42.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:25:42.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=274877906952 2024-09-18T11:25:42.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 274877906952 2024-09-18T11:25:42.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803832 1-322122547203 2-81604378667 3-274877906952' 2024-09-18T11:25:42.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803832 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803832 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803832 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803832' 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803832 2024-09-18T11:25:42.618 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:25:43.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803832 -lt 25769803832 2024-09-18T11:25:43.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:43.092 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-322122547203 2024-09-18T11:25:43.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:43.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:25:43.096 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-322122547203 2024-09-18T11:25:43.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:43.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=322122547203 2024-09-18T11:25:43.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 322122547203' 2024-09-18T11:25:43.099 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 322122547203 2024-09-18T11:25:43.099 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:25:43.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 322122547203 -lt 322122547203 2024-09-18T11:25:43.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:43.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378667 2024-09-18T11:25:43.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:43.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:25:43.547 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:43.547 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378667 2024-09-18T11:25:43.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378667 2024-09-18T11:25:43.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378667' 2024-09-18T11:25:43.549 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378667 2024-09-18T11:25:43.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:25:43.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378667 -lt 81604378667 2024-09-18T11:25:43.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:43.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-274877906952 2024-09-18T11:25:43.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:43.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:25:43.978 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:43.978 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-274877906952 2024-09-18T11:25:43.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=274877906952 2024-09-18T11:25:43.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 274877906952' 2024-09-18T11:25:43.979 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 274877906952 2024-09-18T11:25:43.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:25:44.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 274877906952 -lt 274877906952 2024-09-18T11:25:44.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:25:44.395 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:25:44.396 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:25:45.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:25:45.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:25:45.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:25:45.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:25:45.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:25:45.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:25:45.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:25:45.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:25:45.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:25:45.408 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:25:45.409 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:25:45.409 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:25:45.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:25:45.939 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:25:45.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:25:45.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:25:45.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:25:46.703 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:25:46.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-0-0 fail 2024-09-18T11:25:46.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:25:46.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:25:46.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-0-0 2024-09-18T11:25:46.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:25:46.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:25:46.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79554-0-0 td/test-erasure-eio/COPY 2024-09-18T11:25:46.883 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-size-79554-0-0: (5) Input/output error 2024-09-18T11:25:46.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:25:46.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-09-18T11:25:46.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79554-0-256 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79554-0-256 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79554-0-256 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:46.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:25:46.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:25:46.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:25:46.894 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79554-0-256 td/test-erasure-eio/ORIGINAL 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79554-0-256 td/test-erasure-eio 0 256 add 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-0-256 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-18T11:25:47.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:25:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-18T11:25:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:25:47.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:25:47.086 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-0-256 2024-09-18T11:25:47.086 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:25:47.086 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-0-256 2024-09-18T11:25:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-0-256 2024-09-18T11:25:47.087 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:25:47.517 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:25:47.517 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:47.518 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:25:47.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:25:47.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:25:47.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-09-18T11:25:47.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:25:48.456 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T11:25:48.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:25:48.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-18T11:25:48.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:25:48.471 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:25:48.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:25:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:25:48.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-18T11:25:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:25:51.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:25:51.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:25:51.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:25:51.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T11:25:51.771 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.3 2024-09-18T11:25:51.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:25:51.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-18T11:25:51.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T11:25:51.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:25:51.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:25:51.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:25:51.802 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:51.796+0000 7fc9525c8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:51.839 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:51.836+0000 7fc9525c8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:51.840 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:51.836+0000 7fc9525c8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:25:52.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T11:25:52.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:25:52.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:25:52.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:52.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:52.389 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:52.384+0000 7fc9525c8540 -1 Falling back to public interface 2024-09-18T11:25:52.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:53.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:53.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:53.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:25:53.639 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:53.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:53.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:54.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:55.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:25:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:55.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:55.121 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:25:55.116+0000 7fc9525c8540 -1 osd.3 81 log_to_monitors true 2024-09-18T11:25:55.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:25:56.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:25:56.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:25:56.729 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:25:56.729 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:25:56.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:25:56.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 up in weight 1 up_from 87 up_thru 87 down_at 83 last_clean_interval [64,81) [v2:127.0.0.1:6826/210688,v1:127.0.0.1:6827/210688] [v2:127.0.0.1:6828/210688,v1:127.0.0.1:6829/210688] exists,up 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:25:57.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:25:57.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:25:57.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:25:57.151 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:25:57.151 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:25:57.151 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:25:57.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:25:57.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:25:57.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:25:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:25:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:25:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:25:57.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:25:57.354 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:25:57.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:25:57.839 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:25:57.839 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:25:57.839 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:25:57.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:25:57.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:57.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:25:58.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803837 2024-09-18T11:25:58.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803837 2024-09-18T11:25:58.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837' 2024-09-18T11:25:58.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:58.025 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:25:58.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=322122547208 2024-09-18T11:25:58.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 322122547208 2024-09-18T11:25:58.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837 1-322122547208' 2024-09-18T11:25:58.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:58.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:25:58.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378672 2024-09-18T11:25:58.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378672 2024-09-18T11:25:58.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837 1-322122547208 2-81604378672' 2024-09-18T11:25:58.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:25:58.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:25:58.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=373662154755 2024-09-18T11:25:58.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 373662154755 2024-09-18T11:25:58.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803837 1-322122547208 2-81604378672 3-373662154755' 2024-09-18T11:25:58.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:25:58.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803837 2024-09-18T11:25:58.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:25:58.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:25:58.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803837 2024-09-18T11:25:58.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:25:58.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803837 2024-09-18T11:25:58.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803837' 2024-09-18T11:25:58.464 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803837 2024-09-18T11:25:58.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:25:58.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803835 -lt 25769803837 2024-09-18T11:25:58.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:25:59.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:25:59.868 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:00.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803837 -lt 25769803837 2024-09-18T11:26:00.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:00.326 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-322122547208 2024-09-18T11:26:00.327 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:00.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:26:00.330 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-322122547208 2024-09-18T11:26:00.331 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:00.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=322122547208 2024-09-18T11:26:00.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 322122547208' 2024-09-18T11:26:00.333 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 322122547208 2024-09-18T11:26:00.334 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:26:00.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 322122547208 -lt 322122547208 2024-09-18T11:26:00.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:00.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378672 2024-09-18T11:26:00.787 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:00.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:26:00.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378672 2024-09-18T11:26:00.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:00.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378672 2024-09-18T11:26:00.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378672' 2024-09-18T11:26:00.792 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378672 2024-09-18T11:26:00.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:26:01.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378673 -lt 81604378672 2024-09-18T11:26:01.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:01.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:01.258 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-373662154755 2024-09-18T11:26:01.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:26:01.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-373662154755 2024-09-18T11:26:01.261 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:01.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=373662154755 2024-09-18T11:26:01.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 373662154755' 2024-09-18T11:26:01.264 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 373662154755 2024-09-18T11:26:01.264 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:26:01.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 373662154756 -lt 373662154755 2024-09-18T11:26:01.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:26:01.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:01.709 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:02.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:26:02.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:26:02.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:26:02.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:26:02.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:26:02.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:26:02.224 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:26:02.225 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:26:02.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:26:02.646 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:26:02.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:02.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:26:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:26:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:26:03.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:256 bytes copied, 7.6286e-05 s, 3.4 MB/s 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:03.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:26:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:26:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-09-18T11:26:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:26:03.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-09-18T11:26:03.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:03.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:03.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:03.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:03.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:26:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-09-18T11:26:03.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:06.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-09-18T11:26:06.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:26:06.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:26:06.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:26:06.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:26:06.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:26:06.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:06.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:06.435 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:06.435 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:06.436 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:06.436 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:26:06.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:26:06.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:26:06.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:26:06.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-18T11:26:06.438 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.3 2024-09-18T11:26:06.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:26:06.440 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-09-18T11:26:06.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-18T11:26:06.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:26:06.442 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:26:06.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:26:06.467 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:06.464+0000 7f2e7b253540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:06.471 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:06.468+0000 7f2e7b253540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:06.476 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:06.472+0000 7f2e7b253540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:06.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:06.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:26:07.033 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:07.028+0000 7f2e7b253540 -1 Falling back to public interface 2024-09-18T11:26:07.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:08.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:08.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:08.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:26:08.262 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:08.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:08.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:26:08.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:09.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:09.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:09.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:26:09.657 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:26:09.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:09.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:26:09.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:09.672+0000 7f2e7b253540 -1 osd.3 88 log_to_monitors true 2024-09-18T11:26:10.274 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:11.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:11.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:11.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:26:11.276 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:26:11.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:11.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:26:11.925 INFO:tasks.workunit.client.0.smithi174.stderr:osd.3 up in weight 1 up_from 93 up_thru 93 down_at 89 last_clean_interval [87,88) [v2:127.0.0.1:6826/212113,v1:127.0.0.1:6827/212113] [v2:127.0.0.1:6828/212113,v1:127.0.0.1:6829/212113] exists,up 088c3fab-467e-44cc-bdeb-542cb0a1ca7b 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:26:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:26:11.927 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:26:11.927 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:26:11.928 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:26:11.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:26:11.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:26:11.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:26:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:26:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:26:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:26:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:26:12.135 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:12.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:26:12.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803842 2024-09-18T11:26:12.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803842 2024-09-18T11:26:12.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842' 2024-09-18T11:26:12.775 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:12.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:26:12.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=322122547213 2024-09-18T11:26:12.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 322122547213 2024-09-18T11:26:12.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842 1-322122547213' 2024-09-18T11:26:12.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:12.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:26:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378677 2024-09-18T11:26:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378677 2024-09-18T11:26:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842 1-322122547213 2-81604378677' 2024-09-18T11:26:13.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:13.070 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:26:13.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=399431958531 2024-09-18T11:26:13.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 399431958531 2024-09-18T11:26:13.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803842 1-322122547213 2-81604378677 3-399431958531' 2024-09-18T11:26:13.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:13.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803842 2024-09-18T11:26:13.219 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:13.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:26:13.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:13.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803842 2024-09-18T11:26:13.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803842 2024-09-18T11:26:13.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-09-18T11:26:13.222 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803842 2024-09-18T11:26:13.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:13.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803842 -lt 25769803842 2024-09-18T11:26:13.626 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:13.627 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-322122547213 2024-09-18T11:26:13.628 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:13.630 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:26:13.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-322122547213 2024-09-18T11:26:13.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:13.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=322122547213 2024-09-18T11:26:13.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 322122547213' 2024-09-18T11:26:13.634 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 322122547213 2024-09-18T11:26:13.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:26:14.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 322122547211 -lt 322122547213 2024-09-18T11:26:14.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:26:15.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:26:15.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:26:15.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 322122547214 -lt 322122547213 2024-09-18T11:26:15.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:15.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378677 2024-09-18T11:26:15.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:15.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:26:15.469 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:15.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378677 2024-09-18T11:26:15.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378677 2024-09-18T11:26:15.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378677' 2024-09-18T11:26:15.471 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378677 2024-09-18T11:26:15.472 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:26:15.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378678 -lt 81604378677 2024-09-18T11:26:15.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:15.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-399431958531 2024-09-18T11:26:15.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:15.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:26:15.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-399431958531 2024-09-18T11:26:15.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:15.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=399431958531 2024-09-18T11:26:15.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 399431958531' 2024-09-18T11:26:15.918 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 399431958531 2024-09-18T11:26:15.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:26:16.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 399431958531 -lt 399431958531 2024-09-18T11:26:16.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:26:16.362 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:16.363 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:16.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:26:16.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:26:16.869 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:26:16.869 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:26:16.869 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:26:16.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:26:16.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:26:16.870 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:26:17.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:26:17.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:26:17.285 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:17.285 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:17.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:26:17.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:26:17.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:26:17.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:26:17.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:26:18.512 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-0-256 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-0-256 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:26:18.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79554-0-256 td/test-erasure-eio/COPY 2024-09-18T11:26:18.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:26:18.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:26:18.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79554-0-256 td/test-erasure-eio 1 256 add 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-0-256 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:26:18.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-18T11:26:18.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:26:18.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:26:18.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-0-256 2024-09-18T11:26:18.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:26:18.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-0-256 2024-09-18T11:26:18.726 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-0-256 2024-09-18T11:26:18.726 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:26:19.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:26:19.194 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:19.194 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:26:19.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:26:19.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:26:19.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-18T11:26:19.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:26:20.024 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:26:20.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:26:20.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:26:20.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:20.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:20.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:20.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:20.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:20.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:26:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:26:20.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:26:20.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:26:20.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:26:20.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:23.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:26:23.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:26:23.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:26:23.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:26:23.139 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:26:23.139 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:23.139 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:23.140 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:26:23.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:26:23.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:26:23.142 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:26:23.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:26:23.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:26:23.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:26:23.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:26:23.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:26:23.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:26:23.173 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:23.168+0000 7feadb77a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:23.176 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:23.172+0000 7feadb77a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:23.178 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:23.176+0000 7feadb77a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:23.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:23.725 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:23.720+0000 7feadb77a540 -1 Falling back to public interface 2024-09-18T11:26:24.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:26:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:25.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:25.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:25.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:26.325 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:26.320+0000 7feadb77a540 -1 osd.1 98 log_to_monitors true 2024-09-18T11:26:26.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:26.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:26.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:26:26.484 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:26:26.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:26.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:27.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:28.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:28.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:28.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:26:28.052 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:26:28.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:28.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:28.589 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 104 up_thru 104 down_at 100 last_clean_interval [75,98) [v2:127.0.0.1:6810/213637,v1:127.0.0.1:6811/213637] [v2:127.0.0.1:6812/213637,v1:127.0.0.1:6813/213637] exists,up c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:26:28.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:26:28.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:26:28.590 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:26:28.591 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:26:28.591 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:26:28.591 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:26:28.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:26:28.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:26:28.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:26:28.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:26:28.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:26:28.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:26:28.805 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:29.253 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:26:29.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803847 2024-09-18T11:26:29.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803847 2024-09-18T11:26:29.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803847' 2024-09-18T11:26:29.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:29.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:26:29.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=446676598787 2024-09-18T11:26:29.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 446676598787 2024-09-18T11:26:29.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803847 1-446676598787' 2024-09-18T11:26:29.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:29.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:26:29.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378683 2024-09-18T11:26:29.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378683 2024-09-18T11:26:29.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803847 1-446676598787 2-81604378683' 2024-09-18T11:26:29.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:29.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:26:29.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=399431958536 2024-09-18T11:26:29.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 399431958536 2024-09-18T11:26:29.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803847 1-446676598787 2-81604378683 3-399431958536' 2024-09-18T11:26:29.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:29.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803847 2024-09-18T11:26:29.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:29.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:26:29.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803847 2024-09-18T11:26:29.903 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:29.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803847 2024-09-18T11:26:29.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803847' 2024-09-18T11:26:29.905 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803847 2024-09-18T11:26:29.906 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:30.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803845 -lt 25769803847 2024-09-18T11:26:30.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:26:31.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:26:31.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:31.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803847 -lt 25769803847 2024-09-18T11:26:31.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:31.834 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-446676598787 2024-09-18T11:26:31.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:31.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:26:31.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-446676598787 2024-09-18T11:26:31.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:31.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=446676598787 2024-09-18T11:26:31.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 446676598787' 2024-09-18T11:26:31.840 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 446676598787 2024-09-18T11:26:31.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:26:32.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 446676598787 -lt 446676598787 2024-09-18T11:26:32.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:32.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:32.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378683 2024-09-18T11:26:32.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:26:32.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378683 2024-09-18T11:26:32.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:32.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378683 2024-09-18T11:26:32.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378683' 2024-09-18T11:26:32.245 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378683 2024-09-18T11:26:32.245 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:26:32.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378683 -lt 81604378683 2024-09-18T11:26:32.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:32.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-399431958536 2024-09-18T11:26:32.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:32.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:26:32.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-399431958536 2024-09-18T11:26:32.679 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:32.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=399431958536 2024-09-18T11:26:32.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 399431958536' 2024-09-18T11:26:32.681 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 399431958536 2024-09-18T11:26:32.682 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:26:33.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 399431958537 -lt 399431958536 2024-09-18T11:26:33.121 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:26:33.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:33.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:33.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:26:33.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:26:33.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:26:33.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:26:33.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:26:33.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:26:33.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:26:33.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:26:34.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:26:34.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:26:34.064 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:34.064 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:34.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:26:34.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:26:34.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:26:34.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-18T11:26:34.577 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:26:34.577 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:26:34.578 INFO:tasks.workunit.client.0.smithi174.stderr:256 bytes copied, 7.7786e-05 s, 3.3 MB/s 2024-09-18T11:26:34.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:34.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:26:34.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:26:34.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:26:34.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:26:34.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:34.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:26:34.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:26:34.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:26:34.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:26:34.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:26:34.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:34.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:34.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:34.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:34.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:34.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:26:34.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:26:37.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:26:37.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:26:37.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:26:37.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:37.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:26:37.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:26:37.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:26:37.582 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:26:37.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:26:37.583 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:26:37.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:26:37.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:26:37.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:26:37.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:26:37.612 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:37.607+0000 7ffb60e5b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:37.615 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:37.611+0000 7ffb60e5b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:37.618 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:37.611+0000 7ffb60e5b540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:26:38.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:26:38.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:38.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:26:38.039 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:26:38.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:38.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:38.157 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:38.151+0000 7ffb60e5b540 -1 Falling back to public interface 2024-09-18T11:26:38.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:26:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:39.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:40.729 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:40.723+0000 7ffb60e5b540 -1 osd.1 106 log_to_monitors true 2024-09-18T11:26:40.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:40.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:40.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:26:40.911 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:26:40.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:40.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:41.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:26:42.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:26:42.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:26:42.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:26:42.424 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:26:42.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:26:42.424 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:26:43.150 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 111 up_thru 111 down_at 107 last_clean_interval [104,106) [v2:127.0.0.1:6810/215062,v1:127.0.0.1:6811/215062] [v2:127.0.0.1:6812/215062,v1:127.0.0.1:6813/215062] exists,up c82771f6-0a9f-4710-9729-929630717b79 2024-09-18T11:26:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:26:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:26:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:26:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:26:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:26:43.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:26:43.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:26:43.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:26:43.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:26:43.153 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:26:43.153 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:26:43.154 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:26:43.154 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:26:43.154 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:26:43.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:26:43.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:26:43.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:26:43.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:26:43.368 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:43.768 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:26:43.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803852 2024-09-18T11:26:43.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803852 2024-09-18T11:26:43.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803852' 2024-09-18T11:26:43.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:43.988 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:26:44.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=476741369859 2024-09-18T11:26:44.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 476741369859 2024-09-18T11:26:44.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803852 1-476741369859' 2024-09-18T11:26:44.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:44.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:26:44.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378687 2024-09-18T11:26:44.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378687 2024-09-18T11:26:44.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803852 1-476741369859 2-81604378687' 2024-09-18T11:26:44.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:26:44.372 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:26:44.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=399431958541 2024-09-18T11:26:44.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 399431958541 2024-09-18T11:26:44.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803852 1-476741369859 2-81604378687 3-399431958541' 2024-09-18T11:26:44.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:44.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803852 2024-09-18T11:26:44.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:44.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:26:44.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803852 2024-09-18T11:26:44.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:44.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803852 2024-09-18T11:26:44.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803852' 2024-09-18T11:26:44.525 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803852 2024-09-18T11:26:44.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:44.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803850 -lt 25769803852 2024-09-18T11:26:44.927 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:26:45.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:26:45.930 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:26:46.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803852 -lt 25769803852 2024-09-18T11:26:46.335 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:46.336 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-476741369859 2024-09-18T11:26:46.337 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:46.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:26:46.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-476741369859 2024-09-18T11:26:46.340 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=476741369859 2024-09-18T11:26:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 476741369859' 2024-09-18T11:26:46.342 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 476741369859 2024-09-18T11:26:46.343 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:26:46.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 476741369859 -lt 476741369859 2024-09-18T11:26:46.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:46.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378687 2024-09-18T11:26:46.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:46.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:26:46.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378687 2024-09-18T11:26:46.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:46.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378687 2024-09-18T11:26:46.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378687' 2024-09-18T11:26:46.762 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 81604378687 2024-09-18T11:26:46.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:26:47.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378688 -lt 81604378687 2024-09-18T11:26:47.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:26:47.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-399431958541 2024-09-18T11:26:47.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:26:47.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:26:47.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:26:47.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-399431958541 2024-09-18T11:26:47.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=399431958541 2024-09-18T11:26:47.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 399431958541' 2024-09-18T11:26:47.221 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 399431958541 2024-09-18T11:26:47.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:26:47.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 399431958542 -lt 399431958541 2024-09-18T11:26:47.616 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:26:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:48.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:26:48.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:26:48.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:26:48.203 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:26:48.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:26:48.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:26:48.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:26:48.204 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:26:48.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:26:48.635 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:26:48.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:26:48.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:26:49.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:26:49.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:26:49.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:26:49.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:26:49.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:26:49.903 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:26:49.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-0-256 fail 2024-09-18T11:26:49.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:26:49.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:26:49.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-0-256 2024-09-18T11:26:49.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:26:49.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:26:49.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79554-0-256 td/test-erasure-eio/COPY 2024-09-18T11:26:50.103 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-size-79554-0-256: (5) Input/output error 2024-09-18T11:26:50.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:26:50.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:26:50.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:26:50.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:26:50.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:26:50.695 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:26:50.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:26:51.373 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:26:51.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:26:51.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:26:51.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:26:51.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:26:51.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:51.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:51.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:51.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:51.387 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:51.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:51.505 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:26:51.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:26:51.508 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:26:51.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:26:51.510 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:26:51.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:26:51.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:51.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:26:51.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:26:51.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:51.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:26:51.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:26:51.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:26:51.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:26:51.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:51.549 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:51.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:26:51.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:26:51.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:26:51.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:26:51.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:26:51.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:26:51.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:26:51.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:26:51.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:26:51.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:26:51.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:26:51.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:26:51.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:26:51.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:26:51.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:26:51.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:26:51.558 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:26:51.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:26:51.560 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:26:51.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:26:51.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:26:51.563 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:26:51.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:51.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:26:51.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:26:51.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:26:51.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:26:51.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:26:51.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:26:51.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:26:51.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:51.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:51.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:26:51.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:26:51.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:26:51.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:26:51.576 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:26:51.576 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:51.576 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:51.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:26:51.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:26:51.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:26:51.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:26:51.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:26:51.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:26:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:26:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:26:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:26:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:26:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:26:51.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:26:51.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:26:51.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:51.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:51.989 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:51.989 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:51.990 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:51.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:51.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:26:52.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:26:52.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:26:52.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:26:52.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:26:52.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:26:52.295 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:26:52.295 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:26:52.296 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:26:52.296 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:26:52.297 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:26:52.297 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:52.297 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:52.297 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:26:52.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:26:52.298 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:26:52.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:26:52.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:26:52.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:26:52.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:52.387 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:52.388 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:26:52.388 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:26:52.388 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:26:52.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:26:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:26:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:26:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:26:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:26:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:26:52.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:26:53.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:26:53.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:53.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:53.380 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:53.381 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:53.381 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:53.381 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:53.382 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:26:53.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:26:53.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:26:53.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:26:53.740 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:26:53.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:26:54.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:26:54.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:26:54.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:54.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:26:54.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:26:54.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:26:54.835 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:26:54.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:26:54.845 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:26:52.027+0000 7f7f67074a00 0 load: jerasure load: lrc 2024-09-18T11:26:54.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-09-18T11:26:54.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-09-18T11:26:54.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-09-18T11:26:54.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:26:54.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:26:54.847 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:26:54.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:26:54.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:26:54.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:54.854 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:26:54.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:26:54.856 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:26:54.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=342ead16-9e40-4e72-b467-b384288adc49 2024-09-18T11:26:54.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 342ead16-9e40-4e72-b467-b384288adc49' 2024-09-18T11:26:54.856 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 342ead16-9e40-4e72-b467-b384288adc49 2024-09-18T11:26:54.857 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:26:54.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+uOpmi5X8MxAA9VQ15B3xQGjkKegZE4HvyA== 2024-09-18T11:26:54.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+uOpmi5X8MxAA9VQ15B3xQGjkKegZE4HvyA=="}' 2024-09-18T11:26:54.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 342ead16-9e40-4e72-b467-b384288adc49 -i td/test-erasure-eio/0/new.json 2024-09-18T11:26:55.236 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:26:55.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:26:55.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+uOpmi5X8MxAA9VQ15B3xQGjkKegZE4HvyA== --osd-uuid 342ead16-9e40-4e72-b467-b384288adc49 2024-09-18T11:26:55.284 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.279+0000 7fae6ff59540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:55.288 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.283+0000 7fae6ff59540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:55.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.283+0000 7fae6ff59540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:26:55.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.283+0000 7fae6ff59540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:26:55.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.283+0000 7fae6ff59540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:26:55.290 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.287+0000 7fae6ff59540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:26:55.450 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:26:55.447+0000 7fae6ff59540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:27:01.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:27:01.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:27:01.233 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:27:01.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:27:01.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:27:01.764 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:27:01.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:27:01.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:27:01.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:27:01.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:27:01.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:27:01.792 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:01.787+0000 7f68b8218540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:01.794 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:01.791+0000 7f68b8218540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:01.801 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:01.791+0000 7f68b8218540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:02.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:27:02.156 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:27:02.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:02.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:02.345 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:02.339+0000 7f68b8218540 -1 Falling back to public interface 2024-09-18T11:27:02.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:03.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:03.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:03.580 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:27:03.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:27:03.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:03.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:03.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:04.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:04.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:04.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:27:04.982 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:27:04.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:04.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:05.065 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:05.059+0000 7f68b8218540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:27:05.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:06.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:06.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:06.377 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:27:06.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:27:06.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:06.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:06.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:07.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:07.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:07.966 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:27:07.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:27:07.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:07.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:08.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:09.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:09.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:09.392 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:27:09.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:27:09.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:09.393 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:09.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:10.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:10.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:10.944 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:27:10.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:27:10.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:10.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:11.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:12.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:12.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:12.359 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:27:12.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:27:12.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:12.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:12.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:13.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:13.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:13.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:27:13.851 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:27:13.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:13.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:14.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:15.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:15.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:15.301 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:27:15.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:27:15.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:15.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:15.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:16.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:16.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:16.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:27:16.867 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:27:16.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:16.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:17.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:18.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:18.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:18.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:27:18.315 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:27:18.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:18.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:18.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:19.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:19.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:19.789 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:27:19.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:27:19.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:19.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:20.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:21.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:21.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:21.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:27:21.822 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:27:21.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:21.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:22.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:23.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:23.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:23.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:27:23.512 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:27:23.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:23.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:24.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:25.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:25.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:25.377 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:27:25.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:27:25.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:25.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:25.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:26.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:26.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:26.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:27:26.865 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:27:26.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:26.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:27.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:28.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:28.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:28.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:27:28.822 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:27:28.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:28.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:29.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:30.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:30.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:30.262 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:27:30.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:27:30.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:30.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:27:30.735 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/217358,v1:127.0.0.1:6803/217358] [v2:127.0.0.1:6804/217358,v1:127.0.0.1:6805/217358] exists,up 342ead16-9e40-4e72-b467-b384288adc49 2024-09-18T11:27:30.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:27:30.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:27:30.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:27:30.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:27:30.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:27:30.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:27:30.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:27:30.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:27:30.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:27:30.739 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:27:30.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:27:30.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:27:30.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:27:30.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b2875767-abfb-4ad8-aac2-9f1cc402ab14' 2024-09-18T11:27:30.743 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:27:30.744 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:27:30.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiuepmiNVxLRAAESdlBdg9o2HXJbDVCqPO/g== 2024-09-18T11:27:30.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiuepmiNVxLRAAESdlBdg9o2HXJbDVCqPO/g=="}' 2024-09-18T11:27:30.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2875767-abfb-4ad8-aac2-9f1cc402ab14 -i td/test-erasure-eio/1/new.json 2024-09-18T11:27:31.318 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:27:31.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:27:31.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAiuepmiNVxLRAAESdlBdg9o2HXJbDVCqPO/g== --osd-uuid b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:27:31.368 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.363+0000 7fe1a9319540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:31.392 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.387+0000 7fe1a9319540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:31.394 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.391+0000 7fe1a9319540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:31.395 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.391+0000 7fe1a9319540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:27:31.395 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.391+0000 7fe1a9319540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:27:31.395 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.391+0000 7fe1a9319540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:27:31.484 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:31.479+0000 7fe1a9319540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:27:37.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:27:37.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:27:37.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:27:37.534 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:27:37.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:27:38.097 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:27:38.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:27:38.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:27:38.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:27:38.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:27:38.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:27:38.128 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:38.123+0000 7fb8ba9eb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:38.131 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:38.127+0000 7fb8ba9eb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:38.132 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:38.127+0000 7fb8ba9eb540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:38.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:27:38.540 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:27:38.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:38.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:38.700 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:38.695+0000 7fb8ba9eb540 -1 Falling back to public interface 2024-09-18T11:27:38.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:39.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:39.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:39.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:27:39.943 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:27:39.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:39.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:40.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:41.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:41.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:41.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:27:41.362 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:27:41.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:41.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:41.434 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:27:41.427+0000 7fb8ba9eb540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:27:41.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:42.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:42.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:42.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:27:42.791 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:27:42.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:42.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:43.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:44.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:44.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:44.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:27:44.378 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:27:44.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:44.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:44.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:45.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:45.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:45.807 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:27:45.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:27:45.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:45.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:46.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:47.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:47.339 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:47.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:27:47.340 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:27:47.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:47.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:47.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:48.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:48.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:48.973 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:27:48.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:27:48.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:48.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:49.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:50.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:50.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:50.501 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:27:50.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:27:50.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:50.502 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:50.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:52.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:52.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:52.001 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:27:52.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:27:52.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:52.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:52.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:53.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:53.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:53.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:27:53.573 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:27:53.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:53.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:54.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:27:55.030 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:27:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:55.030 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:56.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:57.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:57.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:57.014 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:27:57.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:27:57.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:57.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:57.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:27:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:27:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:27:58.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:27:58.678 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:27:58.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:27:58.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:27:59.453 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:00.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:00.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:00.455 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:28:00.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:28:00.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:00.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:01.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:02.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:02.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:02.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:28:02.146 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:28:02.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:02.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:02.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:03.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:03.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:03.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:28:03.881 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:28:03.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:03.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:04.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:05.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:05.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:05.380 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:28:05.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:28:05.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:05.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:05.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:06.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:06.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:06.886 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:28:06.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:28:06.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:06.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:28:07.297 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/219159,v1:127.0.0.1:6811/219159] [v2:127.0.0.1:6812/219159,v1:127.0.0.1:6813/219159] exists,up b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:28:07.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:28:07.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:28:07.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:28:07.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:28:07.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:28:07.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:28:07.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7254e1e8-1551-44fc-bbbf-7957bd0a5187' 2024-09-18T11:28:07.306 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:28:07.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:28:07.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHuepmuhZsExAAEHISvJHy5+39ODC05MzdRg== 2024-09-18T11:28:07.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHuepmuhZsExAAEHISvJHy5+39ODC05MzdRg=="}' 2024-09-18T11:28:07.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7254e1e8-1551-44fc-bbbf-7957bd0a5187 -i td/test-erasure-eio/2/new.json 2024-09-18T11:28:07.888 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:28:07.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:28:07.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHuepmuhZsExAAEHISvJHy5+39ODC05MzdRg== --osd-uuid 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:28:07.935 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:07.930+0000 7f728b7d8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:07.938 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:07.930+0000 7f728b7d8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:07.939 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:07.934+0000 7f728b7d8540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:07.939 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:07.934+0000 7f728b7d8540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:28:07.939 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:07.934+0000 7f728b7d8540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:28:07.939 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:07.934+0000 7f728b7d8540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:28:08.009 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:08.002+0000 7f728b7d8540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:28:14.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:28:14.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:28:14.349 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:28:14.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:28:14.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:28:14.919 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:28:14.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:28:14.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:28:14.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:28:14.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:28:14.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:28:14.949 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:14.942+0000 7fbd35848540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:14.952 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:14.946+0000 7fbd35848540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:14.953 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:14.946+0000 7fbd35848540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:15.241 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:15.234+0000 7fbd35848540 -1 Falling back to public interface 2024-09-18T11:28:15.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:28:15.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:28:15.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:28:15.292 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:28:15.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:28:15.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:15.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:28:15.293 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:28:15.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:15.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:15.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:16.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:16.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:16.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:28:16.864 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:28:16.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:16.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:17.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:18.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:28:18.256 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:28:18.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:18.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:18.382 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:18.374+0000 7fbd35848540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:28:18.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:19.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:19.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:19.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:28:19.714 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:28:19.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:19.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:20.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:21.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:21.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:21.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:28:21.198 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:28:21.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:21.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:21.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:22.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:22.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:22.771 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:28:22.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:28:22.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:22.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:23.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:24.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:24.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:24.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:28:24.293 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:28:24.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:24.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:24.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:25.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:25.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:25.700 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:28:25.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:28:25.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:25.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:26.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:27.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:27.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:27.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:28:27.221 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:28:27.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:27.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:27.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:28.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:28.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:28.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:28:28.709 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:28:28.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:28.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:29.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:30.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:30.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:30.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:28:30.236 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:28:30.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:30.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:30.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:31.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:31.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:31.671 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:28:31.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:28:31.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:31.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:32.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:33.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:33.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:33.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:28:33.083 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:28:33.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:33.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:33.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:34.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:34.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:34.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:28:34.500 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:28:34.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:34.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:35.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:36.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:36.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:36.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:28:36.282 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:28:36.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:36.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:36.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:37.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:37.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:37.914 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:28:37.915 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:28:37.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:37.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:38.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:39.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:39.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:39.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:28:39.721 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:28:39.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:39.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:40.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:41.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:41.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:41.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:28:41.573 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:28:41.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:41.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:41.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:42.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:42.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:42.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:28:42.983 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:28:42.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:42.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:43.477 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:44.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:44.479 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:44.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:28:44.480 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-18T11:28:44.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:44.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:28:45.116 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/220960,v1:127.0.0.1:6819/220960] [v2:127.0.0.1:6820/220960,v1:127.0.0.1:6821/220960] exists,up 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:28:45.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:28:45.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:28:45.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:28:45.121 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:28:45.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:28:45.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:28:45.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:28:45.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:28:45.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=226f985f-54a5-41bc-aa52-844b4bc9a4de 2024-09-18T11:28:45.125 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 226f985f-54a5-41bc-aa52-844b4bc9a4de 2024-09-18T11:28:45.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 226f985f-54a5-41bc-aa52-844b4bc9a4de' 2024-09-18T11:28:45.126 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:28:45.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtuepmw+OYCBAAX6r3LlbwqhxF1jj+k7lfjw== 2024-09-18T11:28:45.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtuepmw+OYCBAAX6r3LlbwqhxF1jj+k7lfjw=="}' 2024-09-18T11:28:45.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 226f985f-54a5-41bc-aa52-844b4bc9a4de -i td/test-erasure-eio/3/new.json 2024-09-18T11:28:45.825 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:28:45.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:28:45.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtuepmw+OYCBAAX6r3LlbwqhxF1jj+k7lfjw== --osd-uuid 226f985f-54a5-41bc-aa52-844b4bc9a4de 2024-09-18T11:28:45.864 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:45.858+0000 7fd55e30a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:45.982 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:45.974+0000 7fd55e30a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:45.984 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:45.978+0000 7fd55e30a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:45.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:45.978+0000 7fd55e30a540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:28:45.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:45.978+0000 7fd55e30a540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:28:45.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:45.978+0000 7fd55e30a540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:28:46.134 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:46.130+0000 7fd55e30a540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:28:52.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:28:52.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:28:52.973 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:28:52.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:28:52.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:28:53.510 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:28:53.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:28:53.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:28:53.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:28:53.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:28:53.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:28:53.542 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:53.538+0000 7f9a6c879540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:53.545 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:53.538+0000 7f9a6c879540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:53.547 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:53.542+0000 7f9a6c879540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:53.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:28:53.911 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:28:53.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:53.911 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:28:54.093 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:54.086+0000 7f9a6c879540 -1 Falling back to public interface 2024-09-18T11:28:54.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:55.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:55.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:55.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:28:55.305 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:28:55.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:55.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:28:55.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:56.568 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:28:56.562+0000 7f9a6c879540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:28:56.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:56.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:56.759 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:28:56.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:28:56.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:56.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:28:57.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:28:58.257 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:28:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:28:58.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:28:59.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:28:59.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:28:59.845 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:28:59.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:28:59.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:28:59.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:00.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:01.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:01.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:01.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:29:01.332 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:29:01.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:01.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:01.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:02.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:02.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:02.963 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:29:02.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:29:02.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:02.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:03.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:04.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:04.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:04.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:29:04.375 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:29:04.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:04.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:04.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:05.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:05.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:05.919 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:29:05.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:29:05.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:05.920 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:06.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:07.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:07.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:07.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:29:07.346 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:29:07.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:07.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:07.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:08.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:08.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:08.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:29:08.950 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:29:08.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:08.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:09.365 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:10.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:10.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:10.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:29:10.368 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:29:10.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:10.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:11.152 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:12.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:12.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:12.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:29:12.155 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:29:12.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:12.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:12.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:13.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:29:13.973 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:29:13.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:13.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:14.492 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:15.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:15.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:15.495 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:29:15.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:29:15.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:15.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:15.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:16.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:16.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:16.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:29:16.958 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:29:16.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:16.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:17.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:18.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:18.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:18.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:29:18.667 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:29:18.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:18.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:19.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:20.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:20.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:20.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:29:20.667 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:29:20.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:20.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:21.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:22.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:22.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:22.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:29:22.115 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:29:22.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:22.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:29:22.709 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/222787,v1:127.0.0.1:6827/222787] [v2:127.0.0.1:6828/222787,v1:127.0.0.1:6829/222787] exists,up 226f985f-54a5-41bc-aa52-844b4bc9a4de 2024-09-18T11:29:22.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:29:22.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:29:22.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:29:22.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:29:22.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:29:22.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:29:22.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:29:22.711 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:29:22.712 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:29:22.712 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:29:22.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:29:22.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:29:22.803 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:29:22.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:29:22.818 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:27:02.347+0000 7f68b8218540 0 load: jerasure load: lrc 2024-09-18T11:29:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-09-18T11:29:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:29:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:29:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:29:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:29:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:29:22.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:29:22.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:29:22.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:29:23.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:29:23.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:29:24.914 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:29:24.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:29:25.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:29:25.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:29:25.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:29:25.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:29:25.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:29:25.928 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:29:25.929 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:25.929 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:25.930 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:25.930 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:25.930 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:26.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:29:26.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:29:26.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:29:26.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:26.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:29:26.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:29:26.604 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:29:26.605 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:29:26.605 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:29:26.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:29:26.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:26.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:29:26.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803808 2024-09-18T11:29:26.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803808 2024-09-18T11:29:26.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808' 2024-09-18T11:29:26.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:26.799 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:29:26.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=60129542167 2024-09-18T11:29:26.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 60129542167 2024-09-18T11:29:26.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-60129542167' 2024-09-18T11:29:26.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:26.966 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:29:27.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345942 2024-09-18T11:29:27.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345942 2024-09-18T11:29:27.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-60129542167 2-85899345942' 2024-09-18T11:29:27.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:27.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:29:27.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018889 2024-09-18T11:29:27.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018889 2024-09-18T11:29:27.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-60129542167 2-85899345942 3-128849018889' 2024-09-18T11:29:27.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:27.268 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803808 2024-09-18T11:29:27.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:27.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:29:27.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:27.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803808 2024-09-18T11:29:27.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803808 2024-09-18T11:29:27.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-18T11:29:27.274 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803808 2024-09-18T11:29:27.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:27.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803806 -lt 25769803808 2024-09-18T11:29:27.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:29:28.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:29:28.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:29.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-18T11:29:29.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:29.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-60129542167 2024-09-18T11:29:29.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:29.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:29:29.239 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-60129542167 2024-09-18T11:29:29.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:29.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=60129542167 2024-09-18T11:29:29.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 60129542167' 2024-09-18T11:29:29.242 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 60129542167 2024-09-18T11:29:29.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:29:29.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 60129542168 -lt 60129542167 2024-09-18T11:29:29.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:29.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345942 2024-09-18T11:29:29.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:29.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:29:29.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345942 2024-09-18T11:29:29.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:29.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345942 2024-09-18T11:29:29.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-09-18T11:29:29.677 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 85899345942 2024-09-18T11:29:29.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:29:30.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-18T11:29:30.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:30.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018889 2024-09-18T11:29:30.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:30.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:29:30.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:30.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018889 2024-09-18T11:29:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018889 2024-09-18T11:29:30.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018889' 2024-09-18T11:29:30.124 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 128849018889 2024-09-18T11:29:30.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:29:30.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018889 -lt 128849018889 2024-09-18T11:29:30.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:29:30.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:30.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:31.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:29:31.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:31.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:31.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:31.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:31.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:31.037 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:31.037 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:31.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:29:31.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:31.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:31.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:31.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:29:31.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:29:31.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79554-1-10 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79554-1-10 2024-09-18T11:29:31.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79554-1-10 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:29:31.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79554-1-10 td/test-erasure-eio/ORIGINAL 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79554-1-10 td/test-erasure-eio 1 10 set 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-1-10 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:29:32.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-1-10 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-1-10 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-1-10 2024-09-18T11:29:32.127 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:29:32.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:29:32.546 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:29:32.546 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:29:32.547 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:29:32.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:29:32.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-18T11:29:32.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:29:33.307 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:29:33.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:29:33.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-18T11:29:33.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-18T11:29:33.320 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:29:33.320 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:29:33.321 INFO:tasks.workunit.client.0.smithi174.stderr:10 bytes copied, 0.000190571 s, 52.5 kB/s 2024-09-18T11:29:33.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:33.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:29:33.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:29:33.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:29:33.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:29:33.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:29:33.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:29:33.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:29:34.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:29:34.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:29:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:29:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:29:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:29:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:29:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:29:36.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:29:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:29:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:29:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:29:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:29:36.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:29:36.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:29:36.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:29:36.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:29:36.990 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:29:36.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:29:36.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:29:36.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:29:36.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:29:36.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:29:36.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:29:37.022 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:37.013+0000 7f8743ebe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:37.026 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:37.021+0000 7f8743ebe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:37.028 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:37.021+0000 7f8743ebe540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:37.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:29:37.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:37.400 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:29:37.585 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:37.577+0000 7f8743ebe540 -1 Falling back to public interface 2024-09-18T11:29:37.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:38.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:38.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:38.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:29:38.834 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:29:38.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:38.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:29:39.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:40.185 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:40.173+0000 7f8743ebe540 -1 osd.1 41 log_to_monitors true 2024-09-18T11:29:40.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:40.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:40.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:29:40.232 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:29:40.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:40.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:29:40.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:41.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:41.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:41.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:29:41.841 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:29:41.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:41.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:29:42.325 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 42 last_clean_interval [14,41) [v2:127.0.0.1:6810/224732,v1:127.0.0.1:6811/224732] [v2:127.0.0.1:6812/224732,v1:127.0.0.1:6813/224732] exists,up b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:29:42.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:29:42.327 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:29:42.327 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:42.327 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:42.328 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:42.328 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:42.328 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:42.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:29:42.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:29:42.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:29:42.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:42.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:29:43.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:29:43.056 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:29:43.056 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:29:43.056 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:29:43.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:29:43.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:43.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:29:43.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803813 2024-09-18T11:29:43.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803813 2024-09-18T11:29:43.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813' 2024-09-18T11:29:43.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:43.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:29:43.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495619 2024-09-18T11:29:43.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495619 2024-09-18T11:29:43.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-197568495619' 2024-09-18T11:29:43.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:43.378 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:29:43.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345947 2024-09-18T11:29:43.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345947 2024-09-18T11:29:43.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-197568495619 2-85899345947' 2024-09-18T11:29:43.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:29:43.522 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:29:43.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018894 2024-09-18T11:29:43.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018894 2024-09-18T11:29:43.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803813 1-197568495619 2-85899345947 3-128849018894' 2024-09-18T11:29:43.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:43.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803813 2024-09-18T11:29:43.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:43.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:29:43.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:43.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803813 2024-09-18T11:29:43.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803813 2024-09-18T11:29:43.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-09-18T11:29:43.677 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803813 2024-09-18T11:29:43.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:44.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803811 -lt 25769803813 2024-09-18T11:29:44.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:29:45.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:29:45.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:29:45.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803814 -lt 25769803813 2024-09-18T11:29:45.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:45.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-197568495619 2024-09-18T11:29:45.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:45.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:29:45.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:45.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-197568495619 2024-09-18T11:29:45.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495619 2024-09-18T11:29:45.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 197568495619' 2024-09-18T11:29:45.469 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 197568495619 2024-09-18T11:29:45.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:29:45.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-18T11:29:45.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:45.886 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:45.886 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345947 2024-09-18T11:29:45.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:29:45.888 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345947 2024-09-18T11:29:45.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:45.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345947 2024-09-18T11:29:45.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345947' 2024-09-18T11:29:45.891 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 85899345947 2024-09-18T11:29:45.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:29:46.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345948 -lt 85899345947 2024-09-18T11:29:46.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:29:46.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018894 2024-09-18T11:29:46.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:29:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:29:46.313 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:29:46.313 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018894 2024-09-18T11:29:46.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018894 2024-09-18T11:29:46.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018894' 2024-09-18T11:29:46.314 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018894 2024-09-18T11:29:46.315 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:29:46.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018894 -lt 128849018894 2024-09-18T11:29:46.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:29:46.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:46.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:47.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:29:47.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:29:47.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:29:47.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:29:47.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:29:47.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:29:47.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:29:47.268 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:29:47.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:29:47.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:29:47.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:29:47.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:29:48.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:29:48.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:29:48.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:29:48.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:29:48.157 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:29:48.977 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:29:48.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-1-10 2024-09-18T11:29:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:29:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:29:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-1-10 2024-09-18T11:29:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:29:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:29:48.991 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79554-1-10 td/test-erasure-eio/COPY 2024-09-18T11:29:49.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:29:49.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:29:49.198 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79554-1-10 td/test-erasure-eio 2 10 set 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-1-10 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:29:49.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-1-10 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:29:49.202 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-1-10 2024-09-18T11:29:49.203 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-1-10 2024-09-18T11:29:49.203 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:29:49.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:29:49.651 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:29:49.651 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:29:49.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:29:49.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:29:49.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-18T11:29:49.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:29:50.525 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:29:50.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:29:50.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-09-18T11:29:50.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-09-18T11:29:50.538 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:29:50.539 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:29:50.539 INFO:tasks.workunit.client.0.smithi174.stderr:10 bytes copied, 0.000191791 s, 52.1 kB/s 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:50.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:29:50.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:29:50.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:29:50.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:29:51.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-18T11:29:51.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79554-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:29:54.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-18T11:29:54.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:29:54.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:29:54.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:29:54.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:29:54.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:29:54.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:29:54.284 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:29:54.284 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:29:54.284 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:29:54.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:29:54.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:29:54.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:29:54.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:29:54.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:29:54.286 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.2 2024-09-18T11:29:54.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:29:54.287 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:29:54.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:29:54.289 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:29:54.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:29:54.290 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:29:54.318 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:54.309+0000 7f6abcb22540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:54.321 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:54.313+0000 7f6abcb22540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:54.323 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:54.317+0000 7f6abcb22540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:29:54.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:54.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:29:54.878 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:54.869+0000 7f6abcb22540 -1 Falling back to public interface 2024-09-18T11:29:55.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:56.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:56.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:56.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:29:56.115 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:29:56.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:56.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:29:56.497 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:57.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:57.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:57.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:29:57.499 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:29:57.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:57.500 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:29:57.799 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:29:57.793+0000 7f6abcb22540 -1 osd.2 52 log_to_monitors true 2024-09-18T11:29:58.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:29:59.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:29:59.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:29:59.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:29:59.141 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:29:59.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:29:59.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:29:59.627 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 up in weight 1 up_from 56 up_thru 42 down_at 53 last_clean_interval [20,52) [v2:127.0.0.1:6818/226257,v1:127.0.0.1:6819/226257] [v2:127.0.0.1:6820/226257,v1:127.0.0.1:6821/226257] exists,up 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:29:59.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:29:59.629 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:29:59.630 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:29:59.630 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:29:59.630 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:29:59.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:29:59.631 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:29:59.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:29:59.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:29:59.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:29:59.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:29:59.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:30:00.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:30:00.319 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:00.319 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:00.319 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:30:00.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:30:00.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:00.320 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:30:00.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803819 2024-09-18T11:30:00.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803819 2024-09-18T11:30:00.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819' 2024-09-18T11:30:00.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:00.491 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:30:00.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495625 2024-09-18T11:30:00.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495625 2024-09-18T11:30:00.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-197568495625' 2024-09-18T11:30:00.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:00.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:30:00.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=240518168579 2024-09-18T11:30:00.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 240518168579 2024-09-18T11:30:00.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-197568495625 2-240518168579' 2024-09-18T11:30:00.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:00.783 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:30:00.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018899 2024-09-18T11:30:00.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018899 2024-09-18T11:30:00.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-197568495625 2-240518168579 3-128849018899' 2024-09-18T11:30:00.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:00.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803819 2024-09-18T11:30:00.935 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:00.937 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:30:00.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803819 2024-09-18T11:30:00.938 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:00.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803819 2024-09-18T11:30:00.940 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-09-18T11:30:00.941 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803819 2024-09-18T11:30:00.941 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:30:01.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803819 -lt 25769803819 2024-09-18T11:30:01.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:01.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-197568495625 2024-09-18T11:30:01.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:01.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:30:01.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-197568495625 2024-09-18T11:30:01.356 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:01.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495625 2024-09-18T11:30:01.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 197568495625' 2024-09-18T11:30:01.358 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 197568495625 2024-09-18T11:30:01.358 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:30:01.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495622 -lt 197568495625 2024-09-18T11:30:01.813 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:30:02.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:30:02.815 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:30:03.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495625 -lt 197568495625 2024-09-18T11:30:03.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:03.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:03.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-240518168579 2024-09-18T11:30:03.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:30:03.239 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-240518168579 2024-09-18T11:30:03.239 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:03.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=240518168579 2024-09-18T11:30:03.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 240518168579' 2024-09-18T11:30:03.242 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 240518168579 2024-09-18T11:30:03.242 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:30:03.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 240518168579 -lt 240518168579 2024-09-18T11:30:03.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:03.698 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018899 2024-09-18T11:30:03.699 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:03.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:30:03.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:03.702 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018899 2024-09-18T11:30:03.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018899 2024-09-18T11:30:03.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018899' 2024-09-18T11:30:03.704 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018899 2024-09-18T11:30:03.704 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:30:04.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018900 -lt 128849018899 2024-09-18T11:30:04.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:30:04.178 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:04.178 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:04.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:30:04.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:30:04.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:30:04.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:30:04.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:30:04.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:30:04.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:30:04.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:05.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:30:05.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:30:05.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:05.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:05.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:30:05.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:30:05.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:30:05.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:30:05.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:30:06.360 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:30:06.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-1-10 fail 2024-09-18T11:30:06.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:30:06.373 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:30:06.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-1-10 2024-09-18T11:30:06.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:30:06.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:30:06.374 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79554-1-10 td/test-erasure-eio/COPY 2024-09-18T11:30:06.594 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-size-79554-1-10: (5) Input/output error 2024-09-18T11:30:06.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:30:06.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:30:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:324: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-09-18T11:30:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-18T11:30:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-18T11:30:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-18T11:30:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-18T11:30:06.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79554-1-0 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79554-1-0 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79554-1-0 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:30:06.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:06.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:30:06.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:06.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:30:06.604 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79554-1-0 td/test-erasure-eio/ORIGINAL 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79554-1-0 td/test-erasure-eio 1 0 set 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-1-0 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:30:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-18T11:30:06.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:30:06.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:30:06.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:30:06.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:30:06.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-1-0 2024-09-18T11:30:06.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:30:06.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-1-0 2024-09-18T11:30:06.786 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-1-0 2024-09-18T11:30:06.786 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-18T11:30:07.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:30:08.034 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:30:08.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:30:08.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-18T11:30:08.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-18T11:30:08.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:08.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:30:08.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:30:08.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:08.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:08.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:08.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:08.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:08.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:08.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:08.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:30:08.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:30:08.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:30:08.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:30:08.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:30:08.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:10.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:30:10.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:10.967 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:30:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:30:10.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:30:10.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:30:10.971 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:30:10.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:30:10.972 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:30:10.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:30:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:30:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:30:10.974 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:30:11.001 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:10.996+0000 7efe0dc36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:11.005 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:11.000+0000 7efe0dc36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:11.006 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:11.004+0000 7efe0dc36540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:11.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:30:11.382 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:30:11.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:11.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:11.561 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:11.556+0000 7efe0dc36540 -1 Falling back to public interface 2024-09-18T11:30:11.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:12.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:12.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:12.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:30:12.809 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:12.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:12.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:13.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:14.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:14.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:14.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:30:14.218 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:14.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:14.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:14.372 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:14.368+0000 7efe0dc36540 -1 osd.1 61 log_to_monitors true 2024-09-18T11:30:14.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:15.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:15.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:15.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:30:15.657 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:30:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:16.391 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [46,61) [v2:127.0.0.1:6810/227801,v1:127.0.0.1:6811/227801] [v2:127.0.0.1:6812/227801,v1:127.0.0.1:6813/227801] exists,up b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:30:16.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:30:16.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:30:16.391 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:30:16.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:30:16.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:30:16.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:30:16.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:30:16.392 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:30:16.392 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:30:16.393 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:30:16.393 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:30:16.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:30:16.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:30:16.394 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:30:16.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:30:16.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:30:16.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:30:16.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:30:16.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:30:17.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:30:17.072 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:17.072 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:17.072 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:30:17.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:30:17.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:17.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:30:17.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803824 2024-09-18T11:30:17.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803824 2024-09-18T11:30:17.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824' 2024-09-18T11:30:17.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:17.265 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:30:17.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=287762808835 2024-09-18T11:30:17.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 287762808835 2024-09-18T11:30:17.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-287762808835' 2024-09-18T11:30:17.411 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:17.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:30:17.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=240518168584 2024-09-18T11:30:17.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 240518168584 2024-09-18T11:30:17.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-287762808835 2-240518168584' 2024-09-18T11:30:17.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:17.585 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:30:17.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018905 2024-09-18T11:30:17.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018905 2024-09-18T11:30:17.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-287762808835 2-240518168584 3-128849018905' 2024-09-18T11:30:17.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:17.736 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803824 2024-09-18T11:30:17.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:17.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:30:17.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803824 2024-09-18T11:30:17.740 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:17.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803824 2024-09-18T11:30:17.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-18T11:30:17.743 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803824 2024-09-18T11:30:17.744 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:30:18.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803822 -lt 25769803824 2024-09-18T11:30:18.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:30:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:30:19.151 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:30:19.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803824 -lt 25769803824 2024-09-18T11:30:19.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:19.578 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:19.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-287762808835 2024-09-18T11:30:19.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:30:19.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-287762808835 2024-09-18T11:30:19.581 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:19.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=287762808835 2024-09-18T11:30:19.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2024-09-18T11:30:19.584 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 287762808835 2024-09-18T11:30:19.584 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:30:20.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 287762808835 -lt 287762808835 2024-09-18T11:30:20.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:20.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:20.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-240518168584 2024-09-18T11:30:20.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:30:20.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-240518168584 2024-09-18T11:30:20.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:20.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=240518168584 2024-09-18T11:30:20.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 240518168584' 2024-09-18T11:30:20.020 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 240518168584 2024-09-18T11:30:20.021 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:30:20.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 240518168585 -lt 240518168584 2024-09-18T11:30:20.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:20.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018905 2024-09-18T11:30:20.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:20.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:30:20.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018905 2024-09-18T11:30:20.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:20.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018905 2024-09-18T11:30:20.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018905' 2024-09-18T11:30:20.431 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018905 2024-09-18T11:30:20.431 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:30:20.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018905 -lt 128849018905 2024-09-18T11:30:20.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:30:20.863 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:20.864 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:21.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:30:21.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:30:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:30:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:30:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:30:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:30:21.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:30:21.438 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:21.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:30:21.878 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:30:21.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:21.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:22.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:30:22.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:30:22.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:30:22.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:30:22.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:30:23.097 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-1-0 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-1-0 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:30:23.108 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79554-1-0 td/test-erasure-eio/COPY 2024-09-18T11:30:23.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:30:23.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:30:23.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79554-1-0 td/test-erasure-eio 2 0 set 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-1-0 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-09-18T11:30:23.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:30:23.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:30:23.280 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-1-0 2024-09-18T11:30:23.280 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:30:23.280 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-1-0 2024-09-18T11:30:23.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-1-0 2024-09-18T11:30:23.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:30:23.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:30:23.673 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:23.673 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:30:23.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:30:23.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:30:23.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-18T11:30:23.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:30:24.444 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:30:24.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-09-18T11:30:24.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-09-18T11:30:24.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-09-18T11:30:24.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:24.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:30:24.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:30:24.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:30:24.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:24.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:24.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:24.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:24.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:24.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:24.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:24.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:30:24.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:30:24.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T11:30:24.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:30:24.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-18T11:30:24.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79554-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:27.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:30:27.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:27.603 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:30:27.605 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:30:27.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:30:27.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:30:27.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:30:27.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:30:27.606 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:30:27.606 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.2 2024-09-18T11:30:27.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:30:27.608 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:30:27.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:30:27.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:30:27.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:30:27.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:30:27.641 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:27.636+0000 7f5a97b68540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:27.643 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:27.640+0000 7f5a97b68540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:27.654 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:27.640+0000 7f5a97b68540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:27.932 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:27.928+0000 7f5a97b68540 -1 Falling back to public interface 2024-09-18T11:30:28.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:30:28.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:28.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:30:28.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:29.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:29.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:29.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:30:29.436 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:29.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:30:29.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:29.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:30.583 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:30.580+0000 7f5a97b68540 -1 osd.2 73 log_to_monitors true 2024-09-18T11:30:30.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:30.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:30.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:30:30.847 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:30.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:30.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:30:31.298 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 up in weight 1 up_from 79 up_thru 63 down_at 75 last_clean_interval [56,73) [v2:127.0.0.1:6818/229325,v1:127.0.0.1:6819/229325] [v2:127.0.0.1:6820/229325,v1:127.0.0.1:6821/229325] exists,up 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:30:31.299 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:30:31.300 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:30:31.300 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:30:31.300 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:30:31.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:30:31.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:30:31.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:30:31.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:30:31.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:30:31.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:30:31.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:30:32.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:30:32.051 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:32.051 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:32.051 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:30:32.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:30:32.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:32.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:30:32.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803829 2024-09-18T11:30:32.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803829 2024-09-18T11:30:32.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829' 2024-09-18T11:30:32.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:32.197 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:30:32.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=287762808840 2024-09-18T11:30:32.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 287762808840 2024-09-18T11:30:32.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-287762808840' 2024-09-18T11:30:32.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:32.435 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:30:32.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416387 2024-09-18T11:30:32.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416387 2024-09-18T11:30:32.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-287762808840 2-339302416387' 2024-09-18T11:30:32.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:32.737 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:30:32.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018910 2024-09-18T11:30:32.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018910 2024-09-18T11:30:32.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803829 1-287762808840 2-339302416387 3-128849018910' 2024-09-18T11:30:32.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:32.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803829 2024-09-18T11:30:32.872 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:32.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:30:32.876 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803829 2024-09-18T11:30:32.876 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:32.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803829 2024-09-18T11:30:32.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-09-18T11:30:32.878 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803829 2024-09-18T11:30:32.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:30:33.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803829 -lt 25769803829 2024-09-18T11:30:33.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:33.350 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-287762808840 2024-09-18T11:30:33.350 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:33.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:30:33.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-287762808840 2024-09-18T11:30:33.353 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:33.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=287762808840 2024-09-18T11:30:33.354 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 287762808840' 2024-09-18T11:30:33.355 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 287762808840 2024-09-18T11:30:33.355 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:30:33.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 287762808840 -lt 287762808840 2024-09-18T11:30:33.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:33.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416387 2024-09-18T11:30:33.868 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:33.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:30:33.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:33.871 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416387 2024-09-18T11:30:33.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416387 2024-09-18T11:30:33.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416387' 2024-09-18T11:30:33.872 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 339302416387 2024-09-18T11:30:33.873 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:30:34.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 339302416387 2024-09-18T11:30:34.276 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:30:35.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:30:35.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:30:35.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416387 -lt 339302416387 2024-09-18T11:30:35.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:35.801 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018910 2024-09-18T11:30:35.801 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:35.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:30:35.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:35.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018910 2024-09-18T11:30:35.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018910 2024-09-18T11:30:35.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018910' 2024-09-18T11:30:35.806 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018910 2024-09-18T11:30:35.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:30:36.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018910 -lt 128849018910 2024-09-18T11:30:36.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:30:36.212 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:36.213 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:30:36.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:37.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:30:37.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:30:37.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:37.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:37.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:30:37.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-18T11:30:37.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:30:37.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-18T11:30:37.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:30:37.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:30:37.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:30:37.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:30:37.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:30:37.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:30:37.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:30:37.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:30:37.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:30:37.786 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:38.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:30:38.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:30:38.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:38.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:38.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:30:38.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:30:38.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:30:39.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-18T11:30:39.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-18T11:30:39.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-18T11:30:39.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:30:39.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-18T11:30:39.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:30:39.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:30:39.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:30:39.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:30:39.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:30:39.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:30:39.403 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:30:39.404 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:39.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:30:39.795 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:30:39.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:39.796 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:40.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:30:40.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:30:40.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:30:40.304 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:30:40.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:30:41.104 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:30:41.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-1-0 fail 2024-09-18T11:30:41.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:30:41.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:30:41.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-1-0 2024-09-18T11:30:41.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:30:41.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:30:41.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79554-1-0 td/test-erasure-eio/COPY 2024-09-18T11:30:41.340 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-size-79554-1-0: (5) Input/output error 2024-09-18T11:30:41.344 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:30:41.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:30:41.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-09-18T11:30:41.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-79554-1-256 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-79554-1-256 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-79554-1-256 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:30:41.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-79554-1-256 td/test-erasure-eio/ORIGINAL 2024-09-18T11:30:41.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-79554-1-256 td/test-erasure-eio 1 256 add 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-1-256 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-1-256 2024-09-18T11:30:41.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:30:41.555 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-1-256 2024-09-18T11:30:41.555 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-1-256 2024-09-18T11:30:41.555 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:30:42.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:30:42.058 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:42.058 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:30:42.058 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:30:42.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:30:42.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-09-18T11:30:42.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:30:42.987 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:30:43.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:30:43.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:30:43.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:43.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:43.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:43.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:43.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:43.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:30:43.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:45.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:30:45.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:45.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:30:45.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:30:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:30:45.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:30:45.883 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:30:45.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:30:45.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:30:45.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:30:45.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:30:45.912 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:45.908+0000 7ffa30608540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:45.915 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:45.912+0000 7ffa30608540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:45.917 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:45.912+0000 7ffa30608540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:46.225 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:46.220+0000 7ffa30608540 -1 Falling back to public interface 2024-09-18T11:30:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:30:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:30:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:30:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:30:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:30:46.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:46.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:30:46.312 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:30:46.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:46.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:46.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:47.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:47.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:47.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:30:47.744 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:47.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:47.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:48.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:30:48.788 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:48.784+0000 7ffa30608540 -1 osd.1 83 log_to_monitors true 2024-09-18T11:30:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:30:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:49.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:30:49.175 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:49.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:49.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 90 up_thru 79 down_at 86 last_clean_interval [67,83) [v2:127.0.0.1:6810/230972,v1:127.0.0.1:6811/230972] [v2:127.0.0.1:6812/230972,v1:127.0.0.1:6813/230972] exists,up b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:30:49.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:30:49.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:30:49.721 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:30:49.721 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:30:49.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:30:49.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:30:49.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:30:49.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:30:49.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:30:49.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:30:49.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:30:49.924 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:30:50.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:30:50.386 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:30:50.386 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:30:50.386 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:30:50.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:30:50.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:50.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:30:50.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803835 2024-09-18T11:30:50.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803835 2024-09-18T11:30:50.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835' 2024-09-18T11:30:50.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:50.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:30:50.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=386547056643 2024-09-18T11:30:50.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 386547056643 2024-09-18T11:30:50.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-386547056643' 2024-09-18T11:30:50.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:50.785 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:30:51.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416393 2024-09-18T11:30:51.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416393 2024-09-18T11:30:51.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-386547056643 2-339302416393' 2024-09-18T11:30:51.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:30:51.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:30:51.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018915 2024-09-18T11:30:51.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018915 2024-09-18T11:30:51.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803835 1-386547056643 2-339302416393 3-128849018915' 2024-09-18T11:30:51.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:51.234 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803835 2024-09-18T11:30:51.234 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:51.236 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:30:51.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803835 2024-09-18T11:30:51.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:51.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803835 2024-09-18T11:30:51.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-09-18T11:30:51.239 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803835 2024-09-18T11:30:51.240 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:30:51.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803833 -lt 25769803835 2024-09-18T11:30:51.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:30:52.666 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:30:52.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:30:53.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803835 -lt 25769803835 2024-09-18T11:30:53.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:53.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-386547056643 2024-09-18T11:30:53.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:53.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:30:53.156 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-386547056643 2024-09-18T11:30:53.156 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:53.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=386547056643 2024-09-18T11:30:53.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 386547056643' 2024-09-18T11:30:53.158 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 386547056643 2024-09-18T11:30:53.159 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:30:53.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 386547056643 -lt 386547056643 2024-09-18T11:30:53.588 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:53.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416393 2024-09-18T11:30:53.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:53.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:30:53.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:53.592 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416393 2024-09-18T11:30:53.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416393 2024-09-18T11:30:53.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416393' 2024-09-18T11:30:53.593 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 339302416393 2024-09-18T11:30:53.593 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:30:53.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416393 -lt 339302416393 2024-09-18T11:30:53.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:30:54.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018915 2024-09-18T11:30:54.000 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:30:54.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:30:54.003 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018915 2024-09-18T11:30:54.003 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:30:54.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018915 2024-09-18T11:30:54.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018915' 2024-09-18T11:30:54.005 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018915 2024-09-18T11:30:54.006 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:30:54.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018916 -lt 128849018915 2024-09-18T11:30:54.421 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:30:54.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:54.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:54.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:30:54.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:30:54.945 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:30:54.945 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:30:54.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:30:54.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:30:54.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:30:54.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:30:55.323 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:30:55.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:30:55.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:30:55.324 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:30:55.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:30:55.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:30:55.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:30:55.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:256 bytes copied, 3.4514e-05 s, 7.4 MB/s 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:30:55.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:30:55.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:55.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:30:55.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:30:55.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:30:55.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:30:55.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:30:55.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:30:55.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:30:55.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:30:55.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:30:55.842 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:30:56.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:30:58.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:30:58.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:30:58.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:30:58.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:30:58.955 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:30:58.956 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:30:58.956 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:30:58.956 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:30:58.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:30:58.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:30:58.958 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:30:58.958 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:30:58.959 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:30:58.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:30:58.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:30:58.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:30:58.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:30:58.990 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:58.988+0000 7f81dc7fd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:58.993 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:58.988+0000 7f81dc7fd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:58.995 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:58.992+0000 7f81dc7fd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:30:59.384 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:30:59.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:30:59.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:30:59.557 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:30:59.552+0000 7f81dc7fd540 -1 Falling back to public interface 2024-09-18T11:30:59.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:00.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:00.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:00.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:31:00.832 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:00.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:31:01.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:02.143 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:02.140+0000 7f81dc7fd540 -1 osd.1 91 log_to_monitors true 2024-09-18T11:31:02.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:02.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:02.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:31:02.288 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:31:02.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:02.288 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 96 up_thru 90 down_at 92 last_clean_interval [90,91) [v2:127.0.0.1:6810/232371,v1:127.0.0.1:6811/232371] [v2:127.0.0.1:6812/232371,v1:127.0.0.1:6813/232371] exists,up b2875767-abfb-4ad8-aac2-9f1cc402ab14 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:31:02.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:31:02.820 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:31:02.821 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:31:02.821 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:31:02.821 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:31:02.822 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:31:02.822 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:31:03.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:31:03.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:31:03.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:31:03.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:31:03.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:31:03.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:31:03.513 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:03.513 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:31:03.513 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:31:03.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:31:03.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:03.513 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:31:03.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803839 2024-09-18T11:31:03.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803839 2024-09-18T11:31:03.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839' 2024-09-18T11:31:03.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:03.665 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:31:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860419 2024-09-18T11:31:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860419 2024-09-18T11:31:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-412316860419' 2024-09-18T11:31:03.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:03.830 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:31:04.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=339302416397 2024-09-18T11:31:04.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 339302416397 2024-09-18T11:31:04.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-412316860419 2-339302416397' 2024-09-18T11:31:04.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:04.029 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:31:04.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018920 2024-09-18T11:31:04.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018920 2024-09-18T11:31:04.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803839 1-412316860419 2-339302416397 3-128849018920' 2024-09-18T11:31:04.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:04.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803839 2024-09-18T11:31:04.188 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:04.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:31:04.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803839 2024-09-18T11:31:04.191 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:04.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803839 2024-09-18T11:31:04.193 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803839' 2024-09-18T11:31:04.193 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803839 2024-09-18T11:31:04.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:31:04.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803837 -lt 25769803839 2024-09-18T11:31:04.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:31:05.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:31:05.596 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:31:06.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803840 -lt 25769803839 2024-09-18T11:31:06.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:06.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-412316860419 2024-09-18T11:31:06.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:06.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:31:06.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-412316860419 2024-09-18T11:31:06.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:06.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860419 2024-09-18T11:31:06.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 412316860419' 2024-09-18T11:31:06.067 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 412316860419 2024-09-18T11:31:06.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:31:06.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860419 -lt 412316860419 2024-09-18T11:31:06.560 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:06.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-339302416397 2024-09-18T11:31:06.561 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:06.563 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:31:06.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-339302416397 2024-09-18T11:31:06.565 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:06.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=339302416397 2024-09-18T11:31:06.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 339302416397' 2024-09-18T11:31:06.567 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 339302416397 2024-09-18T11:31:06.567 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:31:07.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 339302416398 -lt 339302416397 2024-09-18T11:31:07.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:07.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018920 2024-09-18T11:31:07.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:07.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:31:07.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018920 2024-09-18T11:31:07.005 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:07.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018920 2024-09-18T11:31:07.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018920' 2024-09-18T11:31:07.007 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018920 2024-09-18T11:31:07.008 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:31:07.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018920 -lt 128849018920 2024-09-18T11:31:07.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:31:07.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:31:07.430 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:31:07.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:31:07.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:31:07.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:31:07.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:31:07.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:31:07.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:31:07.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:31:07.943 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:31:08.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:31:08.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:31:08.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:31:08.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:31:08.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:31:08.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:31:08.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:31:08.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:31:08.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:31:09.608 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-1-256 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-1-256 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:31:09.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-79554-1-256 td/test-erasure-eio/COPY 2024-09-18T11:31:09.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:31:09.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:31:09.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-79554-1-256 td/test-erasure-eio 2 256 add 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-79554-1-256 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-09-18T11:31:09.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:31:09.764 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-79554-1-256 2024-09-18T11:31:09.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:31:09.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-79554-1-256 2024-09-18T11:31:09.765 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-79554-1-256 2024-09-18T11:31:09.765 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:31:10.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:31:10.179 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:10.179 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:31:10.179 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:31:10.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-09-18T11:31:10.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-09-18T11:31:10.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-09-18T11:31:10.980 INFO:tasks.workunit.client.0.smithi174.stderr:noout is set 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:10.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:31:10.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:31:10.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T11:31:10.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:31:10.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:31:10.995 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:10.995 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:10.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:10.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:10.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:11.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-18T11:31:11.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79554-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:14.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-18T11:31:14.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:31:14.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:31:14.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:31:14.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:31:14.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:31:14.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:14.051 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:14.052 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:14.052 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:14.052 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:14.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:31:14.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:31:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:31:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:31:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:31:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:31:14.054 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.2 2024-09-18T11:31:14.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:31:14.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:31:14.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:31:14.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:31:14.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:31:14.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:31:14.085 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:14.080+0000 7fb139116540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:14.088 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:14.084+0000 7fb139116540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:14.091 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:14.088+0000 7fb139116540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:14.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:14.649 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:14.644+0000 7fb139116540 -1 Falling back to public interface 2024-09-18T11:31:14.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:15.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:15.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:15.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:31:15.890 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:15.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:15.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:16.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:17.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:17.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:17.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:31:17.309 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:31:17.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:17.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:17.462 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:17.460+0000 7fb139116540 -1 osd.2 102 log_to_monitors true 2024-09-18T11:31:17.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:18.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:18.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:18.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:31:18.782 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:31:18.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:18.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:19.308 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 up in weight 1 up_from 107 up_thru 92 down_at 103 last_clean_interval [79,102) [v2:127.0.0.1:6818/233869,v1:127.0.0.1:6819/233869] [v2:127.0.0.1:6820/233869,v1:127.0.0.1:6821/233869] exists,up 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:31:19.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:31:19.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:31:19.310 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:31:19.310 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:31:19.310 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:31:19.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:31:19.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:31:19.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:31:19.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:31:19.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:31:19.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:31:19.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:31:19.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:31:20.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:31:20.115 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:20.115 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:31:20.115 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:31:20.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:31:20.116 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:20.116 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:31:20.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803845 2024-09-18T11:31:20.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803845 2024-09-18T11:31:20.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845' 2024-09-18T11:31:20.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:20.256 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:31:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860424 2024-09-18T11:31:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860424 2024-09-18T11:31:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-412316860424' 2024-09-18T11:31:20.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:20.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:31:20.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=459561500675 2024-09-18T11:31:20.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 459561500675 2024-09-18T11:31:20.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-412316860424 2-459561500675' 2024-09-18T11:31:20.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:20.556 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:31:20.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018925 2024-09-18T11:31:20.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018925 2024-09-18T11:31:20.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803845 1-412316860424 2-459561500675 3-128849018925' 2024-09-18T11:31:20.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:20.753 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803845 2024-09-18T11:31:20.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:20.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:31:20.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803845 2024-09-18T11:31:20.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:20.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803845 2024-09-18T11:31:20.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803845' 2024-09-18T11:31:20.759 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803845 2024-09-18T11:31:20.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:31:21.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803845 -lt 25769803845 2024-09-18T11:31:21.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:21.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-412316860424 2024-09-18T11:31:21.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:21.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:31:21.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-412316860424 2024-09-18T11:31:21.219 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:21.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860424 2024-09-18T11:31:21.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 412316860424' 2024-09-18T11:31:21.221 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 412316860424 2024-09-18T11:31:21.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:31:21.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860424 -lt 412316860424 2024-09-18T11:31:21.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:21.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-459561500675 2024-09-18T11:31:21.672 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:21.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:31:21.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-459561500675 2024-09-18T11:31:21.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:21.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=459561500675 2024-09-18T11:31:21.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 459561500675' 2024-09-18T11:31:21.678 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 459561500675 2024-09-18T11:31:21.678 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:31:22.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 459561500675 -lt 459561500675 2024-09-18T11:31:22.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:22.157 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018925 2024-09-18T11:31:22.157 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:22.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:31:22.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018925 2024-09-18T11:31:22.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:22.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018925 2024-09-18T11:31:22.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018925' 2024-09-18T11:31:22.162 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018925 2024-09-18T11:31:22.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:31:22.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018923 -lt 128849018925 2024-09-18T11:31:22.572 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:31:23.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:31:23.574 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:31:23.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018926 -lt 128849018925 2024-09-18T11:31:23.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:31:23.978 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:31:23.978 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:31:24.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:31:24.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:31:24.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:31:24.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:31:24.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:31:24.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:31:24.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:31:24.492 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:31:24.936 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:31:24.937 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:31:24.938 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:31:24.938 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:31:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:31:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:31:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:31:25.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-09-18T11:31:25.454 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records in 2024-09-18T11:31:25.454 INFO:tasks.workunit.client.0.smithi174.stderr:1+0 records out 2024-09-18T11:31:25.454 INFO:tasks.workunit.client.0.smithi174.stderr:256 bytes copied, 8.0219e-05 s, 3.2 MB/s 2024-09-18T11:31:25.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:31:25.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:31:25.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:25.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:25.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:25.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:25.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-18T11:31:25.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-79554-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:31:28.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:28.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:28.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:28.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:28.281 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:28.281 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:28.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:31:28.282 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:31:28.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:31:28.283 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.2 2024-09-18T11:31:28.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:31:28.283 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:31:28.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:31:28.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:31:28.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:31:28.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:31:28.326 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:28.323+0000 7f2d4be19540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:28.330 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:28.327+0000 7f2d4be19540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:28.334 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:28.327+0000 7f2d4be19540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:28.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:31:28.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:31:28.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:31:28.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:31:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:31:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:31:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:31:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:28.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:28.873 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:28.867+0000 7f2d4be19540 -1 Falling back to public interface 2024-09-18T11:31:29.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:30.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:30.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:30.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:31:30.073 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:30.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:30.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:30.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:31.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:31.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:31.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:31:31.474 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:31:31.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:31.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:31.595 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:31.591+0000 7f2d4be19540 -1 osd.2 109 log_to_monitors true 2024-09-18T11:31:31.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:32.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:32.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:32.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:31:32.947 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:31:32.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:32.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 up in weight 1 up_from 114 up_thru 92 down_at 110 last_clean_interval [107,109) [v2:127.0.0.1:6818/235294,v1:127.0.0.1:6819/235294] [v2:127.0.0.1:6820/235294,v1:127.0.0.1:6821/235294] exists,up 7254e1e8-1551-44fc-bbbf-7957bd0a5187 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:31:33.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:31:33.486 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:31:33.486 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:31:33.486 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:31:33.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:31:33.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:31:33.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:31:33.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:31:33.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:31:33.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:31:33.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:31:33.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:31:34.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:31:34.101 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:31:34.101 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:31:34.101 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:31:34.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:31:34.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:34.101 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:31:34.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803849 2024-09-18T11:31:34.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803849 2024-09-18T11:31:34.243 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803849' 2024-09-18T11:31:34.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:34.244 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:31:34.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=412316860429 2024-09-18T11:31:34.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 412316860429 2024-09-18T11:31:34.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803849 1-412316860429' 2024-09-18T11:31:34.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:34.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:31:34.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=489626271747 2024-09-18T11:31:34.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 489626271747 2024-09-18T11:31:34.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803849 1-412316860429 2-489626271747' 2024-09-18T11:31:34.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:31:34.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:31:34.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018930 2024-09-18T11:31:34.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018930 2024-09-18T11:31:34.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803849 1-412316860429 2-489626271747 3-128849018930' 2024-09-18T11:31:34.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:34.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:34.670 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803849 2024-09-18T11:31:34.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:31:34.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803849 2024-09-18T11:31:34.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:34.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803849 2024-09-18T11:31:34.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803849' 2024-09-18T11:31:34.676 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803849 2024-09-18T11:31:34.676 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:31:35.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803850 -lt 25769803849 2024-09-18T11:31:35.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:35.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-412316860429 2024-09-18T11:31:35.052 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:35.055 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:31:35.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-412316860429 2024-09-18T11:31:35.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:35.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=412316860429 2024-09-18T11:31:35.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 412316860429' 2024-09-18T11:31:35.058 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 412316860429 2024-09-18T11:31:35.059 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:31:35.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 412316860429 -lt 412316860429 2024-09-18T11:31:35.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:35.496 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-489626271747 2024-09-18T11:31:35.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:35.499 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:31:35.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:35.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-489626271747 2024-09-18T11:31:35.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=489626271747 2024-09-18T11:31:35.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 489626271747' 2024-09-18T11:31:35.501 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 489626271747 2024-09-18T11:31:35.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:31:35.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 489626271747 -lt 489626271747 2024-09-18T11:31:35.904 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:31:35.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018930 2024-09-18T11:31:35.905 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:31:35.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:31:35.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:31:35.908 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018930 2024-09-18T11:31:35.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018930 2024-09-18T11:31:35.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018930' 2024-09-18T11:31:35.909 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018930 2024-09-18T11:31:35.910 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:31:36.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018928 -lt 128849018930 2024-09-18T11:31:36.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:31:37.326 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:31:37.327 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:31:37.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018930 -lt 128849018930 2024-09-18T11:31:37.749 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:31:37.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:31:37.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:31:38.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:31:38.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:31:38.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:31:38.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:31:38.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:31:38.262 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:31:38.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:31:38.263 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:31:38.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:31:38.693 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:31:38.694 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:31:38.694 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:31:39.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:31:39.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:31:39.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:31:39.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-09-18T11:31:39.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-09-18T11:31:40.001 INFO:tasks.workunit.client.0.smithi174.stderr:noout is unset 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-79554-1-256 fail 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-79554-1-256 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:31:40.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-79554-1-256 td/test-erasure-eio/COPY 2024-09-18T11:31:40.208 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-size-79554-1-256: (5) Input/output error 2024-09-18T11:31:40.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:31:40.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:31:40.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:31:40.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:31:40.214 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:31:40.849 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:31:40.860 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:31:41.527 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:31:41.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:31:41.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:31:41.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:31:41.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:31:41.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:41.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:41.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:41.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:41.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:41.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:41.659 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:31:41.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:31:41.662 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:31:41.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:31:41.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:31:41.665 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:31:41.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:41.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:31:41.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:31:41.668 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:41.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:31:41.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:31:41.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:31:41.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:31:41.705 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:41.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:41.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:31:41.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:31:41.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:31:41.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:31:41.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:31:41.710 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:31:41.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:31:41.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:31:41.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:31:41.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:31:41.714 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:31:41.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:31:41.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:31:41.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:31:41.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:31:41.718 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:31:41.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:41.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:31:41.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:31:41.721 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:31:41.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:31:41.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:31:41.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:31:41.726 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:31:41.726 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:41.726 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:41.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:31:41.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:31:41.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:31:41.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:31:41.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:31:41.731 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:41.732 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:41.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:31:41.734 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:31:41.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:31:41.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:31:41.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:31:41.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:31:41.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:31:41.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:31:41.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:31:41.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:31:41.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:31:41.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:31:42.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:31:42.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:42.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:42.143 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:42.143 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:42.143 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:42.143 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:42.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:31:42.414 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:31:42.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:31:42.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:42.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:31:42.416 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:31:42.417 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:31:42.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:31:42.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:31:42.418 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:31:42.419 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:42.419 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:42.419 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:42.419 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:31:42.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:42.420 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:31:42.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:31:42.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:31:42.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:31:42.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:31:42.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:31:42.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:31:42.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:31:42.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:31:42.513 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:42.513 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:42.513 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:42.514 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:31:42.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:31:42.514 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:31:42.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:31:42.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:31:42.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:31:42.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:31:42.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:31:42.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:31:42.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:31:43.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:31:43.499 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:43.500 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:43.500 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:43.500 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:43.500 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:43.501 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:43.504 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:31:43.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:31:43.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:31:43.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:31:43.855 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:31:43.863 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:31:44.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:31:44.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:31:44.867 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:31:44.867 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:31:44.867 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:44.868 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:44.868 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:31:44.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:31:44.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:31:44.945 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:31:44.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:31:44.956 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:31:42.183+0000 7fe435719a00 0 load: jerasure load: lrc 2024-09-18T11:31:44.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-09-18T11:31:44.956 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:255: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-09-18T11:31:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:256: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-09-18T11:31:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:31:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:31:44.957 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:31:44.958 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:31:44.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:31:44.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:31:44.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:31:44.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:31:44.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:31:44.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:31:44.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:31:44.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:31:44.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:31:44.963 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:31:44.963 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:44.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:31:44.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:31:44.965 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:31:44.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe51d410-e083-4763-a01e-c618626597dc 2024-09-18T11:31:44.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fe51d410-e083-4763-a01e-c618626597dc' 2024-09-18T11:31:44.966 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 fe51d410-e083-4763-a01e-c618626597dc 2024-09-18T11:31:44.966 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:31:44.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAguupmOMiSOhAA2QJ1+02cszdD3tuekJzgrA== 2024-09-18T11:31:44.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAguupmOMiSOhAA2QJ1+02cszdD3tuekJzgrA=="}' 2024-09-18T11:31:44.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe51d410-e083-4763-a01e-c618626597dc -i td/test-erasure-eio/0/new.json 2024-09-18T11:31:45.335 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:31:45.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:31:45.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAguupmOMiSOhAA2QJ1+02cszdD3tuekJzgrA== --osd-uuid fe51d410-e083-4763-a01e-c618626597dc 2024-09-18T11:31:45.382 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.375+0000 7f185c133540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:45.385 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.379+0000 7f185c133540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:45.387 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.383+0000 7f185c133540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.383+0000 7f185c133540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:31:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.383+0000 7f185c133540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:31:45.388 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.383+0000 7f185c133540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:31:45.472 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:45.467+0000 7f185c133540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:31:51.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:31:51.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:31:51.221 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:31:51.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:31:51.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:31:51.779 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:51.803+0000 7fe399e4a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:51.807+0000 7fe399e4a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:51.859 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:51.807+0000 7fe399e4a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:31:52.283 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:31:52.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:52.284 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:31:52.374 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:52.367+0000 7fe399e4a540 -1 Falling back to public interface 2024-09-18T11:31:52.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:53.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:53.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:53.740 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:31:53.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:31:53.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:53.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:31:54.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:55.131 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:31:55.127+0000 7fe399e4a540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:31:55.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:55.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:55.143 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:31:55.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:31:55.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:31:55.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:55.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:56.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:56.558 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:56.559 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:31:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:31:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:56.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:31:57.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:58.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:58.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:58.020 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:31:58.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:31:58.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:58.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:31:58.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:31:59.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:31:59.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:31:59.446 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:31:59.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:31:59.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:31:59.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:00.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:01.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:01.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:01.042 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:32:01.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:32:01.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:01.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:01.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:02.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:02.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:02.474 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:32:02.474 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:32:02.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:02.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:02.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:03.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:03.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:03.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:32:03.963 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:32:03.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:03.963 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:04.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:05.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:05.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:05.380 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:32:05.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:32:05.380 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:05.381 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:05.872 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:06.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:06.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:06.874 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:32:06.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:32:06.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:06.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:07.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:08.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:08.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:08.359 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:32:08.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:32:08.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:08.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:08.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:09.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:09.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:09.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:32:09.844 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:32:09.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:09.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:10.907 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:11.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:11.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:11.909 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:32:11.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:32:11.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:11.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:12.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:13.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:13.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:13.662 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:32:13.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:32:13.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:13.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:14.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:15.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:15.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:15.225 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:32:15.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:32:15.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:15.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:16.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:17.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:17.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:17.050 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:32:17.052 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:32:17.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:17.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:17.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:18.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:18.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:18.612 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:32:18.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:32:18.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:18.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:32:19.149 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/237590,v1:127.0.0.1:6803/237590] [v2:127.0.0.1:6804/237590,v1:127.0.0.1:6805/237590] exists,up fe51d410-e083-4763-a01e-c618626597dc 2024-09-18T11:32:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:32:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:32:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:32:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:32:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:32:19.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:32:19.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:32:19.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:32:19.152 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:32:19.153 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:32:19.153 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:32:19.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:32:19.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:32:19.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:32:19.155 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:32:19.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=647c9726-b535-4103-a530-9a84efff00ea 2024-09-18T11:32:19.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 647c9726-b535-4103-a530-9a84efff00ea' 2024-09-18T11:32:19.156 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 647c9726-b535-4103-a530-9a84efff00ea 2024-09-18T11:32:19.156 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:32:19.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDuupm8XhrChAAkNP8uS1zRk2YAyOXhEimgw== 2024-09-18T11:32:19.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDuupm8XhrChAAkNP8uS1zRk2YAyOXhEimgw=="}' 2024-09-18T11:32:19.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 647c9726-b535-4103-a530-9a84efff00ea -i td/test-erasure-eio/1/new.json 2024-09-18T11:32:19.817 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:32:19.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:32:19.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDuupm8XhrChAAkNP8uS1zRk2YAyOXhEimgw== --osd-uuid 647c9726-b535-4103-a530-9a84efff00ea 2024-09-18T11:32:19.865 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.859+0000 7f364febf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:19.869 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.863+0000 7f364febf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:19.871 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.867+0000 7f364febf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:19.873 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.867+0000 7f364febf540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:32:19.873 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.867+0000 7f364febf540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:32:19.873 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.867+0000 7f364febf540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:32:19.975 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:19.971+0000 7f364febf540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:32:26.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:32:26.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:32:26.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:32:26.073 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:32:26.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:32:26.617 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:32:26.617 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:32:26.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:32:26.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:32:26.618 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:32:26.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:32:26.649 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:26.643+0000 7fc9a88b4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:26.677 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:26.671+0000 7fc9a88b4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:26.678 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:26.675+0000 7fc9a88b4540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:27.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:32:27.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:32:27.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:32:27.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:32:27.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:32:27.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:27.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:32:27.076 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:32:27.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:27.076 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:27.232 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:27.227+0000 7fc9a88b4540 -1 Falling back to public interface 2024-09-18T11:32:27.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:28.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:28.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:28.445 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:32:28.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:32:28.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:28.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:28.859 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:29.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:29.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:29.861 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:32:29.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:32:29.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:29.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:29.895 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:29.891+0000 7fc9a88b4540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:32:30.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:31.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:31.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:31.406 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:32:31.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:32:31.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:31.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:31.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:32.822 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:32.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:32.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:32:32.823 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:32:32.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:32.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:33.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:34.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:34.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:34.261 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:32:34.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:32:34.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:34.262 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:34.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:35.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:35.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:35.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:32:35.714 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:32:35.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:35.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:36.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:37.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:37.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:37.166 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:32:37.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:32:37.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:37.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:37.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:38.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:38.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:38.736 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:32:38.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:32:38.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:38.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:39.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:40.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:40.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:40.177 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:32:40.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:32:40.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:40.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:40.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:41.711 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:41.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:41.712 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:32:41.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:32:41.712 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:41.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:42.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:43.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:43.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:43.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:32:43.199 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:32:43.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:43.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:43.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:44.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:44.949 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:44.950 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:32:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:32:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:44.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:45.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:46.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:46.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:46.883 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:32:46.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:32:46.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:46.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:47.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:48.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:48.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:48.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:32:48.765 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:32:48.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:48.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:49.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:50.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:50.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:50.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:32:50.548 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:32:50.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:50.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:51.181 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:52.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:52.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:52.183 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:52.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:52.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:32:53.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:32:53.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:32:53.722 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:32:53.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:32:53.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:32:53.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:32:54.165 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/239367,v1:127.0.0.1:6811/239367] [v2:127.0.0.1:6812/239367,v1:127.0.0.1:6813/239367] exists,up 647c9726-b535-4103-a530-9a84efff00ea 2024-09-18T11:32:54.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:32:54.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:32:54.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:32:54.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:32:54.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:32:54.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:32:54.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:32:54.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:32:54.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:32:54.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:32:54.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:32:54.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:32:54.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30d9b91d-83f3-45b8-850f-df98fcb918d6 2024-09-18T11:32:54.173 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 30d9b91d-83f3-45b8-850f-df98fcb918d6 2024-09-18T11:32:54.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 30d9b91d-83f3-45b8-850f-df98fcb918d6' 2024-09-18T11:32:54.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:32:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmuupm6n2BCxAADgACzHytxedvMrjy3JluJw== 2024-09-18T11:32:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmuupm6n2BCxAADgACzHytxedvMrjy3JluJw=="}' 2024-09-18T11:32:54.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30d9b91d-83f3-45b8-850f-df98fcb918d6 -i td/test-erasure-eio/2/new.json 2024-09-18T11:32:55.134 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:32:55.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:32:55.155 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmuupm6n2BCxAADgACzHytxedvMrjy3JluJw== --osd-uuid 30d9b91d-83f3-45b8-850f-df98fcb918d6 2024-09-18T11:32:55.181 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.174+0000 7f4bd49f1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:55.191 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.186+0000 7f4bd49f1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:55.193 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.186+0000 7f4bd49f1540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:32:55.194 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.190+0000 7f4bd49f1540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:32:55.194 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.190+0000 7f4bd49f1540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:32:55.194 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.190+0000 7f4bd49f1540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:32:55.261 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:32:55.254+0000 7f4bd49f1540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:33:01.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:33:01.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:33:01.737 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:33:01.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:33:01.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:33:02.293 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:33:02.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:33:02.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:33:02.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:33:02.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:33:02.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:33:02.319 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:02.314+0000 7f0147fe6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:02.321 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:02.314+0000 7f0147fe6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:02.322 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:02.318+0000 7f0147fe6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:02.685 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:33:02.686 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:33:02.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:02.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:02.869 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:02.862+0000 7f0147fe6540 -1 Falling back to public interface 2024-09-18T11:33:03.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:04.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:04.126 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:04.127 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:33:04.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:33:04.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:04.127 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:04.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:05.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:05.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:05.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:33:05.549 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:33:05.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:05.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:05.678 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:05.670+0000 7f0147fe6540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:33:06.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:07.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:07.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:07.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:33:07.031 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:33:07.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:07.032 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:07.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:08.439 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:08.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:08.440 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:33:08.440 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:33:08.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:08.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:08.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:10.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:10.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:10.001 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:33:10.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:33:10.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:10.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:10.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:11.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:11.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:11.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:33:11.448 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:33:11.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:11.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:12.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:13.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:13.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:13.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:33:13.144 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:33:13.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:13.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:13.667 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:14.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:14.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:14.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:33:14.671 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:33:14.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:14.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:15.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:16.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:16.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:16.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:33:16.148 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:33:16.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:16.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:16.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:17.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:17.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:17.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:33:17.659 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:33:17.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:17.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:18.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:19.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:19.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:19.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:33:19.187 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:33:19.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:19.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:19.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:20.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:20.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:20.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:33:20.743 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:33:20.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:20.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:21.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:22.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:22.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:22.709 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:33:22.710 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:33:22.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:22.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:23.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:24.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:24.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:24.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:33:24.550 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:33:24.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:24.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:25.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:26.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:26.312 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:26.313 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:33:26.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:33:26.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:26.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:27.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:28.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:28.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:28.169 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:33:28.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:33:28.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:28.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:29.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:30.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:30.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:30.133 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:33:30.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:33:30.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:30.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:30.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:31.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:31.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:31.602 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:33:31.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:33:31.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:31.603 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:33:32.065 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/241141,v1:127.0.0.1:6819/241141] [v2:127.0.0.1:6820/241141,v1:127.0.0.1:6821/241141] exists,up 30d9b91d-83f3-45b8-850f-df98fcb918d6 2024-09-18T11:33:32.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:33:32.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:33:32.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:33:32.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:33:32.074 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:33:32.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e282ccb4-826f-4bdf-a26f-b51699a5b8a2 2024-09-18T11:33:32.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e282ccb4-826f-4bdf-a26f-b51699a5b8a2' 2024-09-18T11:33:32.075 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 e282ccb4-826f-4bdf-a26f-b51699a5b8a2 2024-09-18T11:33:32.075 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:33:32.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMuupmVAN2BRAA898vFVTEWxxkvzai3ju9+Q== 2024-09-18T11:33:32.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMuupmVAN2BRAA898vFVTEWxxkvzai3ju9+Q=="}' 2024-09-18T11:33:32.094 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e282ccb4-826f-4bdf-a26f-b51699a5b8a2 -i td/test-erasure-eio/3/new.json 2024-09-18T11:33:32.693 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:33:32.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:33:32.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMuupmVAN2BRAA898vFVTEWxxkvzai3ju9+Q== --osd-uuid e282ccb4-826f-4bdf-a26f-b51699a5b8a2 2024-09-18T11:33:32.735 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.730+0000 7f138fb3e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:32.751 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.746+0000 7f138fb3e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:32.752 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.746+0000 7f138fb3e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:32.753 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.746+0000 7f138fb3e540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:33:32.753 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.746+0000 7f138fb3e540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:33:32.753 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.746+0000 7f138fb3e540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:33:32.947 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:32.942+0000 7f138fb3e540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:33:39.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:33:39.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:33:39.493 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:33:39.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:33:39.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:33:40.088 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:33:40.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:33:40.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:33:40.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:33:40.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:33:40.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:33:40.119 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:40.114+0000 7ff26741a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:40.121 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:40.114+0000 7ff26741a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:40.122 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:40.118+0000 7ff26741a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:40.493 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:33:40.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:33:40.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:40.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:40.672 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:40.666+0000 7ff26741a540 -1 Falling back to public interface 2024-09-18T11:33:40.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:41.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:41.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:41.923 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:33:41.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:33:41.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:41.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:42.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:43.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:43.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:43.337 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:33:43.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:33:43.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:43.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:43.442 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:33:43.438+0000 7ff26741a540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:33:43.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:44.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:44.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:44.801 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:33:44.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:33:44.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:44.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:45.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:46.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:46.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:46.437 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:33:46.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:33:46.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:46.438 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:46.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:47.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:47.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:47.964 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:33:47.964 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:33:47.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:47.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:48.509 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:49.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:49.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:49.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:33:49.512 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:33:49.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:49.512 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:50.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:51.018 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:51.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:51.019 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:33:51.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:33:51.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:51.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:52.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:52.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:52.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:33:52.611 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:33:52.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:52.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:53.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:54.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:54.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:54.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:33:54.062 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:33:54.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:54.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:54.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:55.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:55.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:55.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:33:55.649 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:33:55.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:55.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:56.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:57.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:57.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:57.124 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:33:57.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:33:57.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:57.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:57.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:33:58.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:33:58.584 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:33:58.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:33:58.585 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:33:58.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:33:58.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:33:59.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:00.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:00.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:00.224 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:34:00.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:34:00.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:00.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:00.980 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:01.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:01.982 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:01.982 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:34:01.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:34:01.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:01.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:02.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:03.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:03.888 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:03.888 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:34:03.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:34:03.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:03.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:04.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:05.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:05.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:05.366 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:34:05.367 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:34:05.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:05.367 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:05.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:06.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:06.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:06.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:34:06.817 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:34:06.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:06.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:07.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:08.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:08.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:08.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:34:08.961 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:34:08.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:08.962 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:09.529 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:10.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:10.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:10.532 INFO:tasks.workunit.client.0.smithi174.stdout:19 2024-09-18T11:34:10.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-18T11:34:10.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:10.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:34:11.159 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/242942,v1:127.0.0.1:6827/242942] [v2:127.0.0.1:6828/242942,v1:127.0.0.1:6829/242942] exists,up e282ccb4-826f-4bdf-a26f-b51699a5b8a2 2024-09-18T11:34:11.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:34:11.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:34:11.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:34:11.160 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:34:11.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:34:11.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:34:11.162 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:11.162 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:11.162 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:11.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:34:11.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:34:11.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:34:11.252 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:34:11.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:34:11.267 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:31:52.375+0000 7fe399e4a540 0 load: jerasure load: lrc 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:259: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:34:11.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:34:12.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:34:12.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:34:13.640 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:34:13.653 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:34:14.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:34:14.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:34:14.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:34:14.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:34:14.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:34:14.657 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:34:14.657 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:34:14.657 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:34:14.658 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:34:14.658 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:34:14.659 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:34:14.882 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:34:14.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:34:14.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:34:14.883 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:34:14.883 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:34:15.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:34:15.285 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:34:15.285 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:34:15.286 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:34:15.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:34:15.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:34:15.286 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:34:15.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803809 2024-09-18T11:34:15.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803809 2024-09-18T11:34:15.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809' 2024-09-18T11:34:15.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:34:15.453 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:34:15.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574877 2024-09-18T11:34:15.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574877 2024-09-18T11:34:15.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-55834574877' 2024-09-18T11:34:15.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:34:15.638 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:34:15.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378646 2024-09-18T11:34:15.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378646 2024-09-18T11:34:15.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-55834574877 2-81604378646' 2024-09-18T11:34:15.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:34:15.844 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:34:15.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051600 2024-09-18T11:34:15.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051600 2024-09-18T11:34:15.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-55834574877 2-81604378646 3-124554051600' 2024-09-18T11:34:15.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:34:15.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803809 2024-09-18T11:34:15.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:34:15.992 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:34:15.994 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803809 2024-09-18T11:34:15.994 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:34:15.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803809 2024-09-18T11:34:15.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-18T11:34:15.996 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803809 2024-09-18T11:34:15.997 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:34:16.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803807 -lt 25769803809 2024-09-18T11:34:16.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:34:17.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:34:17.426 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:34:17.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803809 -lt 25769803809 2024-09-18T11:34:17.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:34:17.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574877 2024-09-18T11:34:17.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:34:17.881 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:34:17.883 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:34:17.883 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574877 2024-09-18T11:34:17.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574877 2024-09-18T11:34:17.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-09-18T11:34:17.885 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574877 2024-09-18T11:34:17.885 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:34:18.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574877 -lt 55834574877 2024-09-18T11:34:18.270 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:34:18.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378646 2024-09-18T11:34:18.271 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:34:18.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:34:18.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378646 2024-09-18T11:34:18.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:34:18.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378646 2024-09-18T11:34:18.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-09-18T11:34:18.275 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378646 2024-09-18T11:34:18.276 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:34:18.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378647 -lt 81604378646 2024-09-18T11:34:18.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:34:18.704 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051600 2024-09-18T11:34:18.704 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:34:18.706 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:34:18.707 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051600 2024-09-18T11:34:18.708 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:34:18.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051600 2024-09-18T11:34:18.710 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051600' 2024-09-18T11:34:18.710 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051600 2024-09-18T11:34:18.711 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:34:19.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051600 -lt 124554051600 2024-09-18T11:34:19.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:34:19.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:34:19.186 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:34:19.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:34:19.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:34:19.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:34:19.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:34:19.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:34:19.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:34:19.745 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:34:19.746 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:34:20.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:34:20.161 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:34:20.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:34:20.162 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:34:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:34:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:34:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:34:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:261: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-09-18T11:34:20.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79554-0 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79554-0 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:34:20.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79554-0 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:34:20.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79554-0 td/test-erasure-eio/ORIGINAL 2024-09-18T11:34:20.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79554-0 td/test-erasure-eio 0 2024-09-18T11:34:20.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:34:20.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:34:20.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:34:20.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-79554-0 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-79554-0 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:34:20.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-0 2024-09-18T11:34:20.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-0 2024-09-18T11:34:20.849 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:34:21.305 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:34:21.306 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:34:21.306 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:34:21.306 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:34:21.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:34:21.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T11:34:21.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:34:21.307 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T11:34:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:34:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T11:34:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:34:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T11:34:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:34:21.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:34:21.311 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:34:21.311 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T11:34:21.311 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:34:21.311 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:34:21.312 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:21.312 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:21.312 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:21.312 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:34:21.312 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:34:21.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:34:21.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:34:21.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:34:21.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T11:34:21.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:34:21.406 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:34:21.406 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:21.406 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:21.406 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:21.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:34:21.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:34:21.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-79554-0 0 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79554-0 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79554-0 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:34:21.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79554-0 td/test-erasure-eio/COPY 2024-09-18T11:34:21.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:34:21.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:34:21.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-18T11:34:21.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=1 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79554-0 td/test-erasure-eio 1 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-79554-0 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:34:21.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:34:21.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:34:21.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T11:34:21.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:34:21.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:34:21.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-79554-0 2024-09-18T11:34:21.553 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:34:21.554 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-0 2024-09-18T11:34:21.554 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-0 2024-09-18T11:34:21.554 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:34:21.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:34:21.950 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:34:21.950 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:34:21.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:34:21.950 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:34:21.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T11:34:21.951 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:34:21.951 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-18T11:34:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:34:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T11:34:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:34:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T11:34:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:34:21.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:34:21.955 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:34:21.955 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T11:34:21.955 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:34:21.955 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:34:21.956 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:21.956 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:21.956 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:21.956 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:34:21.957 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:34:22.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:34:22.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:34:22.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:34:22.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T11:34:22.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:34:22.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:34:22.045 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:22.045 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:22.045 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:22.045 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:34:22.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:34:22.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-79554-0 1 2024-09-18T11:34:22.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79554-0 fail 2024-09-18T11:34:22.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:34:22.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:34:22.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79554-0 2024-09-18T11:34:22.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:34:22.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:34:22.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-79554-0 td/test-erasure-eio/COPY 2024-09-18T11:34:22.185 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-eio-79554-0: (5) Input/output error 2024-09-18T11:34:22.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:34:22.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:34:22.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:263: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:34:22.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:34:22.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:34:22.787 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:34:22.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:34:23.504 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:34:23.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:34:23.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:34:23.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:34:23.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:34:23.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:23.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:23.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:23.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:23.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:23.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:23.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:23.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:23.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:23.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:23.641 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:23.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:23.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:23.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:23.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:23.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:23.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:23.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:34:23.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:34:23.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:23.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:23.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:23.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:34:23.674 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:34:23.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:34:23.675 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:34:23.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:34:23.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:34:23.675 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:34:23.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:34:23.681 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:34:23.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:34:23.683 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:34:23.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:34:23.686 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:34:23.686 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:34:23.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:23.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:34:23.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:34:23.689 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:34:23.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:34:23.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:34:23.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:34:23.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:34:23.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:23.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:23.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:34:23.697 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:34:23.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:34:23.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:34:23.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:34:23.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:23.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:23.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:34:23.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:34:23.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:34:23.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:34:23.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:34:24.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:34:24.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:24.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:24.104 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:24.104 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:24.105 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:24.105 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:34:24.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:34:24.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:34:24.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:34:24.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:34:24.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:34:24.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:34:24.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:34:24.386 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:34:24.386 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:24.386 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:24.386 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:24.387 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:24.387 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:24.387 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:34:24.388 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:34:24.388 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:34:24.494 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:34:24.494 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:34:24.494 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:34:24.494 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:34:24.495 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:34:24.495 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:34:24.495 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:24.495 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:24.496 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:24.496 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:24.496 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:24.497 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:34:24.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:34:24.497 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:34:24.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:34:25.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:34:25.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:25.518 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:25.519 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:25.519 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:25.519 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:25.519 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:34:25.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:34:25.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:34:25.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:34:25.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:34:25.891 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:34:25.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:34:26.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:34:26.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:34:26.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:34:26.901 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:34:26.901 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:26.901 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:26.901 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:34:26.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:34:26.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:34:26.978 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:34:26.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:34:26.989 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:34:24.141+0000 7f0d156b5a00 0 load: jerasure load: lrc 2024-09-18T11:34:26.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-09-18T11:34:26.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:267: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-09-18T11:34:26.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:268: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-09-18T11:34:26.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:34:26.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:34:26.991 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:34:26.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:34:26.993 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:34:26.994 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:34:26.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:34:26.995 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:34:26.996 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:34:26.996 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:34:26.996 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:34:26.996 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:34:26.996 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:34:26.996 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:34:26.997 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:34:26.999 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:34:27.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=330ddffa-23d6-44fe-b89f-996285809f9d 2024-09-18T11:34:27.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 330ddffa-23d6-44fe-b89f-996285809f9d' 2024-09-18T11:34:27.001 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 330ddffa-23d6-44fe-b89f-996285809f9d 2024-09-18T11:34:27.001 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:34:27.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDuupmdRYeARAAZe7xNF1eknOrqL9lHuP16A== 2024-09-18T11:34:27.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDuupmdRYeARAAZe7xNF1eknOrqL9lHuP16A=="}' 2024-09-18T11:34:27.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 330ddffa-23d6-44fe-b89f-996285809f9d -i td/test-erasure-eio/0/new.json 2024-09-18T11:34:27.388 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:34:27.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:34:27.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDuupmdRYeARAAZe7xNF1eknOrqL9lHuP16A== --osd-uuid 330ddffa-23d6-44fe-b89f-996285809f9d 2024-09-18T11:34:27.436 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.429+0000 7fa523f55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:27.439 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.433+0000 7fa523f55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:27.441 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.433+0000 7fa523f55540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:27.441 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.433+0000 7fa523f55540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:34:27.441 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.433+0000 7fa523f55540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:34:27.442 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.433+0000 7fa523f55540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:34:27.545 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:27.537+0000 7fa523f55540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:34:33.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:34:33.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:34:33.733 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:34:33.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:34:33.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:34:34.317 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:34:34.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:34:34.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:34:34.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:34:34.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:34:34.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:34:34.346 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:34.337+0000 7f4b2a467540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:34.348 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:34.341+0000 7f4b2a467540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:34.350 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:34.345+0000 7f4b2a467540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:34:34.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:34:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:34:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:34:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:34:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:34:34.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:34.715 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:34:34.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:34:34.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:34.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:34.920 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:34.913+0000 7f4b2a467540 -1 Falling back to public interface 2024-09-18T11:34:35.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:36.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:36.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:36.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:34:36.141 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:34:36.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:36.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:36.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:37.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:37.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:37.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:34:37.550 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:34:37.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:37.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:37.642 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:34:37.633+0000 7f4b2a467540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:34:38.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:39.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:39.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:39.011 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:34:39.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:34:39.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:39.011 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:39.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:40.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:40.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:40.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:34:40.469 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:34:40.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:40.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:40.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:41.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:41.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:41.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:34:41.908 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:34:41.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:41.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:42.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:43.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:43.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:43.433 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:34:43.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:34:43.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:43.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:43.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:44.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:44.851 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:34:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:34:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:44.851 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:45.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:46.363 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:34:46.364 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:34:46.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:46.364 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:46.786 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:47.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:47.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:47.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:34:47.789 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:34:47.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:47.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:48.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:49.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:49.357 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:49.358 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:34:49.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:34:49.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:49.358 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:50.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:51.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:51.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:51.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:34:51.024 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:34:51.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:51.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:51.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:52.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:52.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:52.737 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:34:52.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:34:52.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:52.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:53.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:54.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:54.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:54.539 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:34:54.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:34:54.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:54.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:55.501 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:56.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:56.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:56.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:34:56.504 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:34:56.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:56.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:57.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:34:58.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:34:58.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:34:58.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:34:58.337 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:34:58.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:34:58.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:34:59.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:00.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:00.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:00.409 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:35:00.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:35:00.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:00.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:35:00.966 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/245808,v1:127.0.0.1:6803/245808] [v2:127.0.0.1:6804/245808,v1:127.0.0.1:6805/245808] exists,up 330ddffa-23d6-44fe-b89f-996285809f9d 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:35:00.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:00.969 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:00.971 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:00.971 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:00.971 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:35:00.971 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:35:00.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:35:00.973 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:35:00.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=458d1c5f-18b0-4e6b-97d6-4bd98494c2d7 2024-09-18T11:35:00.973 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 458d1c5f-18b0-4e6b-97d6-4bd98494c2d7 2024-09-18T11:35:00.973 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 458d1c5f-18b0-4e6b-97d6-4bd98494c2d7' 2024-09-18T11:35:00.974 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:35:00.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkuupm9KVBOxAA+wqJGyGEYMtPm4kI/zvrog== 2024-09-18T11:35:00.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkuupm9KVBOxAA+wqJGyGEYMtPm4kI/zvrog=="}' 2024-09-18T11:35:00.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 458d1c5f-18b0-4e6b-97d6-4bd98494c2d7 -i td/test-erasure-eio/1/new.json 2024-09-18T11:35:01.639 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:35:01.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:35:01.660 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkuupm9KVBOxAA+wqJGyGEYMtPm4kI/zvrog== --osd-uuid 458d1c5f-18b0-4e6b-97d6-4bd98494c2d7 2024-09-18T11:35:01.693 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.689+0000 7fbc98fa6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:01.720 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.717+0000 7fbc98fa6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:01.722 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.721+0000 7fbc98fa6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:01.723 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.721+0000 7fbc98fa6540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:35:01.723 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.721+0000 7fbc98fa6540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:35:01.723 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.721+0000 7fbc98fa6540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:35:01.811 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:01.809+0000 7fbc98fa6540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:35:08.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:35:08.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:35:08.401 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:35:08.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:35:08.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:35:08.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:35:08.966 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:35:08.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:35:08.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:35:08.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:35:08.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:35:08.997 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:08.992+0000 7ffba12a0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:08.999 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:08.996+0000 7ffba12a0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:09.002 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:09.000+0000 7ffba12a0540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:09.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:35:09.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:35:09.346 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:09.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:09.552 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:09.548+0000 7ffba12a0540 -1 Falling back to public interface 2024-09-18T11:35:09.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:10.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:10.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:10.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:10.870 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:35:10.870 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:10.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:11.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:12.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:12.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:12.267 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:35:12.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:35:12.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:12.268 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:12.383 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:12.380+0000 7ffba12a0540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:35:12.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:13.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:13.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:13.751 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:35:13.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:35:13.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:13.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:14.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:15.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:15.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:15.234 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:35:15.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:35:15.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:15.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:15.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:16.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:16.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:16.692 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:35:16.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:35:16.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:16.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:17.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:18.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:18.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:18.237 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:35:18.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:35:18.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:18.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:18.661 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:19.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:19.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:19.663 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:35:19.664 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:35:19.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:19.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:20.151 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:21.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:21.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:21.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:35:21.153 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:35:21.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:21.154 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:21.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:22.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:22.624 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:22.624 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:35:22.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:35:22.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:22.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:23.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:24.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:24.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:24.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:35:24.212 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:35:24.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:24.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:24.625 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:25.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:25.627 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:25.628 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:35:25.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:35:25.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:25.628 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:26.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:27.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:27.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:27.797 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:35:27.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:35:27.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:27.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:28.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:29.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:29.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:29.404 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:35:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:35:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:29.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:30.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:31.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:31.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:31.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:35:31.280 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:35:31.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:31.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:31.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:32.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:32.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:32.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:35:32.820 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:35:32.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:32.821 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:33.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:34.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:34.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:34.753 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:35:34.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:35:34.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:34.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:35.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:36.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:36.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:36.555 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:35:36.555 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:35:36.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:36.556 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:37.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:38.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:38.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:38.105 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:35:38.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:35:38.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:38.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:35:38.640 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/247560,v1:127.0.0.1:6811/247560] [v2:127.0.0.1:6812/247560,v1:127.0.0.1:6813/247560] exists,up 458d1c5f-18b0-4e6b-97d6-4bd98494c2d7 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:35:38.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:35:38.643 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:35:38.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:35:38.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:35:38.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:35:38.644 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:35:38.644 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:35:38.645 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:35:38.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:35:38.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:35:38.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:35:38.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:35:38.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:35:38.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:35:38.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:35:38.647 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:35:38.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aff9db2a-5e84-4f37-8ed0-6c64cf9e16db 2024-09-18T11:35:38.649 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 aff9db2a-5e84-4f37-8ed0-6c64cf9e16db' 2024-09-18T11:35:38.649 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 aff9db2a-5e84-4f37-8ed0-6c64cf9e16db 2024-09-18T11:35:38.650 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:35:38.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAKu+pmvKfVJxAAxc5s5R408pXJA2B93WG1Pg== 2024-09-18T11:35:38.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAKu+pmvKfVJxAAxc5s5R408pXJA2B93WG1Pg=="}' 2024-09-18T11:35:38.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aff9db2a-5e84-4f37-8ed0-6c64cf9e16db -i td/test-erasure-eio/2/new.json 2024-09-18T11:35:39.235 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:35:39.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:35:39.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAKu+pmvKfVJxAAxc5s5R408pXJA2B93WG1Pg== --osd-uuid aff9db2a-5e84-4f37-8ed0-6c64cf9e16db 2024-09-18T11:35:39.282 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.276+0000 7f8e27f07540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:39.285 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.280+0000 7f8e27f07540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:39.286 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.284+0000 7f8e27f07540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:39.287 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.284+0000 7f8e27f07540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:35:39.287 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.284+0000 7f8e27f07540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:35:39.287 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.284+0000 7f8e27f07540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:35:39.357 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:39.352+0000 7f8e27f07540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:35:45.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:35:45.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:35:45.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:35:45.817 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:35:45.817 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:35:46.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:35:46.377 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:35:46.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:35:46.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:35:46.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:35:46.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:35:46.401 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:46.396+0000 7f3f24a16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:46.404 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:46.400+0000 7f3f24a16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:46.406 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:46.400+0000 7f3f24a16540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:35:46.801 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:35:46.802 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:35:46.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:46.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:46.956 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:46.952+0000 7f3f24a16540 -1 Falling back to public interface 2024-09-18T11:35:47.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:48.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:48.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:48.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:35:48.218 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:35:48.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:48.218 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:48.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:49.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:49.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:49.616 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:35:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:35:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:49.822 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:35:49.816+0000 7f3f24a16540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:35:50.017 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:51.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:51.020 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:35:51.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:35:51.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:51.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:51.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:52.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:52.485 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:52.486 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:35:52.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:35:52.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:52.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:53.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:54.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:54.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:54.074 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:35:54.074 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:35:54.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:54.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:54.487 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:55.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:55.489 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:55.490 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:35:55.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:35:55.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:55.490 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:56.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:57.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:57.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:57.174 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:35:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:35:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:57.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:57.727 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:35:58.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:35:58.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:35:58.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:35:58.730 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:35:58.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:35:58.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:35:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:00.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:00.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:00.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:36:00.172 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:36:00.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:00.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:00.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:01.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:01.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:01.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:36:01.751 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:36:01.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:01.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:02.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:03.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:03.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:03.209 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:36:03.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:36:03.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:03.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:03.718 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:04.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:04.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:04.721 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:36:04.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:36:04.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:04.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:05.688 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:06.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:06.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:06.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:36:06.691 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:36:06.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:06.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:07.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:08.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:08.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:08.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:36:08.659 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:36:08.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:08.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:09.713 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:10.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:36:10.716 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:36:10.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:10.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:11.311 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:12.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:12.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:12.313 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:36:12.314 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:36:12.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:12.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:13.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:14.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:14.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:14.007 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:36:14.007 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:36:14.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:14.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:36:14.541 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/249361,v1:127.0.0.1:6819/249361] [v2:127.0.0.1:6820/249361,v1:127.0.0.1:6821/249361] exists,up aff9db2a-5e84-4f37-8ed0-6c64cf9e16db 2024-09-18T11:36:14.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:36:14.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:36:14.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:36:14.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:36:14.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:36:14.547 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:36:14.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:36:14.549 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:36:14.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:36:14.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e8609f3-abbc-4227-be96-5dc279d02a9b 2024-09-18T11:36:14.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7e8609f3-abbc-4227-be96-5dc279d02a9b' 2024-09-18T11:36:14.550 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 7e8609f3-abbc-4227-be96-5dc279d02a9b 2024-09-18T11:36:14.550 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:36:14.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuu+pmr57YIRAAvdLNg0F7JllIcyvLTQEMcg== 2024-09-18T11:36:14.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuu+pmr57YIRAAvdLNg0F7JllIcyvLTQEMcg=="}' 2024-09-18T11:36:14.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e8609f3-abbc-4227-be96-5dc279d02a9b -i td/test-erasure-eio/3/new.json 2024-09-18T11:36:15.326 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:36:15.345 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:36:15.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuu+pmr57YIRAAvdLNg0F7JllIcyvLTQEMcg== --osd-uuid 7e8609f3-abbc-4227-be96-5dc279d02a9b 2024-09-18T11:36:15.379 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.375+0000 7f69aa5aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:15.392 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.387+0000 7f69aa5aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:15.393 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.387+0000 7f69aa5aa540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:15.394 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.391+0000 7f69aa5aa540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:36:15.394 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.391+0000 7f69aa5aa540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:36:15.394 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.391+0000 7f69aa5aa540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:36:15.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:15.527+0000 7f69aa5aa540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:36:22.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:36:22.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:36:22.585 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:36:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:36:22.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:36:23.118 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:36:23.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:36:23.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:36:23.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:36:23.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:36:23.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:36:23.148 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:23.143+0000 7fc295907540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:23.185 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:23.179+0000 7fc295907540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:23.186 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:23.183+0000 7fc295907540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:36:23.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:36:23.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:36:23.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:36:23.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:36:23.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:36:23.503 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:23.504 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:36:23.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:36:23.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:23.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:23.731 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:23.727+0000 7fc295907540 -1 Falling back to public interface 2024-09-18T11:36:23.900 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:24.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:24.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:24.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:36:24.902 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:36:24.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:24.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:25.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:26.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:26.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:26.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:36:26.299 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:36:26.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:26.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:26.421 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:36:26.415+0000 7fc295907540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:36:26.766 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:27.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:27.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:27.768 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:36:27.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:36:27.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:27.769 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:28.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:29.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:29.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:29.460 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:36:29.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:36:29.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:29.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:29.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:30.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:30.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:30.944 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:36:30.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:36:30.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:30.945 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:31.460 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:32.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:32.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:32.463 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:36:32.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:36:32.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:32.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:32.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:33.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:33.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:33.985 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:36:33.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:36:33.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:33.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:34.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:35.451 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:35.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:35.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:36:35.452 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:36:35.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:35.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:36.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:37.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:37.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:37.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:36:37.023 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:36:37.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:37.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:37.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:38.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:38.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:38.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:36:38.600 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:36:38.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:38.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:39.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:40.047 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:40.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:40.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:36:40.048 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:36:40.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:40.048 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:40.493 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:41.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:41.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:41.495 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:36:41.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:36:41.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:41.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:42.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:43.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:43.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:43.464 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:36:43.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:36:43.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:43.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:43.855 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:44.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:44.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:44.857 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:36:44.857 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:36:44.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:44.858 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:46.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:47.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:47.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:47.024 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:36:47.025 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:36:47.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:47.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:48.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:49.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:49.117 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:49.117 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:36:49.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:36:49.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:49.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:49.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:50.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:50.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:50.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:36:50.700 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:36:50.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:50.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:51.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:36:52.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:36:52.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:36:52.722 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:36:52.723 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:36:52.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:36:52.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:36:53.175 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/251137,v1:127.0.0.1:6827/251137] [v2:127.0.0.1:6828/251137,v1:127.0.0.1:6829/251137] exists,up 7e8609f3-abbc-4227-be96-5dc279d02a9b 2024-09-18T11:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:36:53.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:36:53.177 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:36:53.177 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:36:53.177 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:36:53.178 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:36:53.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:36:53.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:36:53.257 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:36:53.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:36:53.271 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:34:34.921+0000 7f4b2a467540 0 load: jerasure load: lrc 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:270: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:271: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:36:53.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:36:53.273 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:36:54.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:36:54.491 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:36:55.761 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:36:55.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:36:56.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:36:56.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:36:56.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:36:56.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:36:56.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:36:56.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:36:56.774 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:36:56.774 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:36:56.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:36:56.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:36:56.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:36:56.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:36:56.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:36:56.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:36:56.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:36:56.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:36:57.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:36:57.427 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:36:57.427 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:36:57.427 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:36:57.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:36:57.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:36:57.428 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:36:57.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803812 2024-09-18T11:36:57.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803812 2024-09-18T11:36:57.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812' 2024-09-18T11:36:57.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:36:57.612 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:36:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607575 2024-09-18T11:36:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607575 2024-09-18T11:36:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-51539607575' 2024-09-18T11:36:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:36:57.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:36:57.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=77309411352 2024-09-18T11:36:57.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 77309411352 2024-09-18T11:36:57.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-51539607575 2-77309411352' 2024-09-18T11:36:57.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:36:57.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:36:58.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084297 2024-09-18T11:36:58.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084297 2024-09-18T11:36:58.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-51539607575 2-77309411352 3-120259084297' 2024-09-18T11:36:58.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:36:58.053 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803812 2024-09-18T11:36:58.054 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:36:58.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:36:58.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803812 2024-09-18T11:36:58.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:36:58.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803812 2024-09-18T11:36:58.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-18T11:36:58.060 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803812 2024-09-18T11:36:58.060 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:36:58.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803812 2024-09-18T11:36:58.461 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:36:59.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:36:59.464 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:36:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-18T11:36:59.912 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:36:59.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-51539607575 2024-09-18T11:36:59.913 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:36:59.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:36:59.916 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-51539607575 2024-09-18T11:36:59.917 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:36:59.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607575 2024-09-18T11:36:59.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 51539607575' 2024-09-18T11:36:59.919 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 51539607575 2024-09-18T11:36:59.919 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:37:00.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607576 -lt 51539607575 2024-09-18T11:37:00.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:37:00.376 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-77309411352 2024-09-18T11:37:00.377 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:37:00.379 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:37:00.380 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-77309411352 2024-09-18T11:37:00.381 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:37:00.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=77309411352 2024-09-18T11:37:00.383 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 77309411352' 2024-09-18T11:37:00.383 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 77309411352 2024-09-18T11:37:00.384 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:37:00.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 77309411352 -lt 77309411352 2024-09-18T11:37:00.771 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:37:00.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084297 2024-09-18T11:37:00.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:37:00.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:37:00.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084297 2024-09-18T11:37:00.776 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:37:00.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084297 2024-09-18T11:37:00.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084297' 2024-09-18T11:37:00.777 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084297 2024-09-18T11:37:00.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:37:01.235 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084297 -lt 120259084297 2024-09-18T11:37:01.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:37:01.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:37:01.237 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:37:01.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:37:01.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:37:01.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:37:01.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:37:01.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:37:01.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:37:01.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:37:01.760 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:37:02.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:37:02.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:37:02.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:37:02.177 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:273: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:274: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-18T11:37:02.678 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79554-1 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79554-1 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:37:02.679 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79554-1 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:37:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79554-1 td/test-erasure-eio/ORIGINAL 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79554-1 td/test-erasure-eio 1 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-79554-1 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:37:02.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-79554-1 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-1 2024-09-18T11:37:02.874 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-1 2024-09-18T11:37:02.875 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:37:03.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:37:03.278 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:37:03.278 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:37:03.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:37:03.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:37:03.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T11:37:03.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:37:03.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-18T11:37:03.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:37:03.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T11:37:03.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:37:03.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T11:37:03.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:37:03.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:37:03.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:37:03.282 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T11:37:03.283 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:37:03.283 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:37:03.284 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:03.284 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:03.284 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:03.284 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:37:03.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:37:03.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:37:03.368 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:37:03.369 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:37:03.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T11:37:03.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:37:03.370 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:37:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:03.370 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:03.371 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:03.371 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:37:03.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:37:03.371 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-79554-1 1 2024-09-18T11:37:03.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79554-1 2024-09-18T11:37:03.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:37:03.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:37:03.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79554-1 2024-09-18T11:37:03.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:37:03.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:37:03.466 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79554-1 td/test-erasure-eio/COPY 2024-09-18T11:37:03.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:37:03.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:37:03.516 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-18T11:37:03.516 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-18T11:37:03.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-09-18T11:37:03.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79554-1 td/test-erasure-eio 2 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-79554-1 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=2 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:37:03.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:37:03.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-79554-1 2024-09-18T11:37:03.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:37:03.520 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-1 2024-09-18T11:37:03.520 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-1 2024-09-18T11:37:03.520 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:37:03.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:37:03.985 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:37:03.985 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:37:03.985 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:37:03.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:37:03.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=2 2024-09-18T11:37:03.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:37:03.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/2/type 2024-09-18T11:37:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:37:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-18T11:37:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:37:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-18T11:37:03.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:37:03.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:37:03.990 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:37:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-18T11:37:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T11:37:03.991 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T11:37:03.992 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:03.992 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:03.992 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:03.992 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.2.asok 2024-09-18T11:37:03.993 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:37:04.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:37:04.086 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:37:04.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:37:04.088 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.2 2024-09-18T11:37:04.088 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-18T11:37:04.088 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-18T11:37:04.089 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:04.089 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:04.089 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:04.089 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.2.asok 2024-09-18T11:37:04.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:37:04.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-79554-1 2 2024-09-18T11:37:04.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79554-1 fail 2024-09-18T11:37:04.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:37:04.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:37:04.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79554-1 2024-09-18T11:37:04.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:37:04.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:37:04.186 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-79554-1 td/test-erasure-eio/COPY 2024-09-18T11:37:04.221 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-eio-79554-1: (5) Input/output error 2024-09-18T11:37:04.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:37:04.224 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:37:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:37:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:37:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:37:04.896 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:37:04.908 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:37:05.585 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:37:05.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:37:05.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:37:05.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:05.597 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:37:05.598 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:05.598 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:05.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:05.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:05.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:05.717 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:05.718 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:05.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:05.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:05.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:05.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:05.722 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:05.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:05.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:05.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:05.725 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:05.726 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:05.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:05.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:37:05.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:05.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:05.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:05.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:37:05.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:05.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:05.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:37:05.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:37:05.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:37:05.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:37:05.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:37:05.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:37:05.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:37:05.753 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:37:05.754 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:37:05.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:37:05.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:37:05.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:37:05.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:37:05.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:37:05.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:37:05.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:37:05.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:37:05.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:37:05.763 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:37:05.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:05.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:37:05.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:37:05.766 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:37:05.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:37:05.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:37:05.770 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:37:05.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:37:05.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:05.772 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:05.773 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:37:05.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:37:05.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:37:05.774 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:37:05.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:37:05.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:05.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:05.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:37:05.780 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:37:05.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:37:05.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:37:05.781 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:37:05.782 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:37:06.194 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:37:06.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:06.195 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:06.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:06.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:06.195 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:06.196 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:37:06.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:37:06.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:37:06.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:37:06.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:06.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:06.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:37:06.544 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:37:06.544 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:06.544 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:06.544 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:06.545 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:06.545 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:06.545 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:06.545 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:37:06.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:06.546 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:37:06.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:37:06.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:37:06.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:37:06.639 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:37:06.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:37:06.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:37:06.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:06.640 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:06.641 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:06.641 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:06.641 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:06.641 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:37:06.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:37:06.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:37:06.728 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:37:07.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:37:07.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:07.634 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:07.634 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:07.634 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:07.635 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:07.635 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:37:07.637 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:07.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:37:07.669 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:37:07.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:37:08.007 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:37:08.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:37:09.017 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:37:09.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:37:09.018 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:37:09.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:37:09.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:09.018 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:09.019 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:37:09.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:37:09.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:37:09.093 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:37:09.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:37:09.103 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:37:06.235+0000 7f461754aa00 0 load: jerasure load: lrc 2024-09-18T11:37:09.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-09-18T11:37:09.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-09-18T11:37:09.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_missing: setup_osds 4 2024-09-18T11:37:09.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:37:09.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:37:09.105 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:37:09.107 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:37:09.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:37:09.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:37:09.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:37:09.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:37:09.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:09.110 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:37:09.112 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:37:09.113 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:37:09.114 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:37:09.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f81680d-ab07-4f20-a936-16104cb1e3d7 2024-09-18T11:37:09.115 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1f81680d-ab07-4f20-a936-16104cb1e3d7' 2024-09-18T11:37:09.115 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 1f81680d-ab07-4f20-a936-16104cb1e3d7 2024-09-18T11:37:09.115 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:37:09.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlu+pm0iLzBxAAWd48RB9A2uwzI/L1iNyxpg== 2024-09-18T11:37:09.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlu+pm0iLzBxAAWd48RB9A2uwzI/L1iNyxpg=="}' 2024-09-18T11:37:09.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f81680d-ab07-4f20-a936-16104cb1e3d7 -i td/test-erasure-eio/0/new.json 2024-09-18T11:37:09.476 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:37:09.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:37:09.488 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlu+pm0iLzBxAAWd48RB9A2uwzI/L1iNyxpg== --osd-uuid 1f81680d-ab07-4f20-a936-16104cb1e3d7 2024-09-18T11:37:09.522 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.515+0000 7f639a288540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:09.530 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.527+0000 7f639a288540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:09.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.527+0000 7f639a288540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:09.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.527+0000 7f639a288540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:37:09.532 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.527+0000 7f639a288540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:37:09.533 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.527+0000 7f639a288540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:37:09.577 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:09.571+0000 7f639a288540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:37:15.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:37:15.359 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:37:15.362 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:37:15.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:37:15.362 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:37:15.941 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:37:15.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:37:15.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:37:15.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:37:15.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:37:15.942 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:37:15.968 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:15.963+0000 7f3032f00540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:15.971 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:15.967+0000 7f3032f00540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:15.972 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:15.967+0000 7f3032f00540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:16.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:37:16.353 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:37:16.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:16.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:16.524 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:16.519+0000 7f3032f00540 -1 Falling back to public interface 2024-09-18T11:37:16.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:17.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:17.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:17.743 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:37:17.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:37:17.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:17.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:18.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:19.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:19.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:19.138 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:37:19.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:37:19.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:19.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:19.248 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:19.242+0000 7f3032f00540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:37:19.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:20.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:20.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:20.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:37:20.543 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:37:20.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:20.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:21.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:22.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:22.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:22.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:37:22.104 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:37:22.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:22.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:22.576 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:23.578 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:37:23.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:37:23.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:23.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:24.075 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:25.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:25.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:25.077 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:37:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:37:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:25.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:25.528 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:26.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:26.530 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:26.531 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:37:26.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:37:26.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:26.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:27.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:28.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:28.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:28.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:37:28.071 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:37:28.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:28.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:28.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:29.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:29.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:29.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:37:29.523 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:37:29.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:29.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:30.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:31.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:31.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:31.064 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:37:31.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:37:31.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:31.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:31.494 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:32.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:32.495 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:32.496 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:37:32.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:37:32.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:32.496 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:33.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:34.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:34.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:34.002 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:37:34.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:37:34.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:34.003 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:35.023 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:36.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:36.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:36.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:37:36.026 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:37:36.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:36.027 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:36.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:37.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:37.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:37.593 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:37:37.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:37:37.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:37.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:38.199 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:39.201 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:39.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:39.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:37:39.202 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:37:39.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:39.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:39.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:40.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:40.969 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:37:40.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:37:40.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:40.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:42.010 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:43.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:43.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:43.012 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:37:43.012 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:37:43.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:43.013 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:43.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:44.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:44.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:44.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:37:44.425 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:37:44.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:44.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:37:44.831 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/253976,v1:127.0.0.1:6803/253976] [v2:127.0.0.1:6804/253976,v1:127.0.0.1:6805/253976] exists,up 1f81680d-ab07-4f20-a936-16104cb1e3d7 2024-09-18T11:37:44.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:37:44.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:37:44.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:37:44.835 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:37:44.837 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:37:44.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:37:44.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:37:44.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=481520d0-06bc-43a8-9997-f1d9ad1b0624 2024-09-18T11:37:44.839 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 481520d0-06bc-43a8-9997-f1d9ad1b0624' 2024-09-18T11:37:44.839 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 481520d0-06bc-43a8-9997-f1d9ad1b0624 2024-09-18T11:37:44.840 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:37:44.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIu+pmmS0uMxAAOn0aMbrYORfCwGz5z5py7Q== 2024-09-18T11:37:44.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIu+pmmS0uMxAAOn0aMbrYORfCwGz5z5py7Q=="}' 2024-09-18T11:37:44.865 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 481520d0-06bc-43a8-9997-f1d9ad1b0624 -i td/test-erasure-eio/1/new.json 2024-09-18T11:37:45.419 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:37:45.434 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:37:45.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIu+pmmS0uMxAAOn0aMbrYORfCwGz5z5py7Q== --osd-uuid 481520d0-06bc-43a8-9997-f1d9ad1b0624 2024-09-18T11:37:45.468 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.462+0000 7f887fff6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:45.470 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.466+0000 7f887fff6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:45.472 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.466+0000 7f887fff6540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:45.473 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.466+0000 7f887fff6540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:37:45.474 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.466+0000 7f887fff6540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:37:45.474 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.466+0000 7f887fff6540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:37:45.580 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:45.574+0000 7f887fff6540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:37:52.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:37:52.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:37:52.133 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:37:52.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:37:52.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:37:52.667 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:37:52.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:37:52.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:37:52.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:37:52.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:37:52.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:37:52.699 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:52.694+0000 7fb8de958540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:52.701 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:52.694+0000 7fb8de958540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:52.703 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:52.698+0000 7fb8de958540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:37:53.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:37:53.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:37:53.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:53.164 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:37:53.271 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:53.266+0000 7fb8de958540 -1 Falling back to public interface 2024-09-18T11:37:53.608 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:37:54.610 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:37:54.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:54.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:37:55.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:56.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:56.041 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:37:56.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:37:56.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:56.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:37:56.141 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:37:56.134+0000 7fb8de958540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:37:56.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:57.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:57.526 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:57.526 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:37:57.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:37:57.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:57.527 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:37:58.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:37:59.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:37:59.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:37:59.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:37:59.058 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:37:59.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:37:59.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:37:59.480 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:00.482 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:38:00.482 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:38:00.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:00.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:01.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:02.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:02.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:02.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:38:02.172 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:38:02.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:02.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:02.598 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:03.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:03.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:03.600 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:38:03.600 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:38:03.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:03.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:04.200 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:05.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:05.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:05.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:38:05.202 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:38:05.202 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:05.203 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:05.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:06.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:06.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:06.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:38:06.707 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:38:06.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:06.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:07.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:08.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:08.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:08.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:38:08.321 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:38:08.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:08.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:08.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:09.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:38:09.790 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:38:09.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:09.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:10.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:11.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:11.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:11.355 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:38:11.356 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:38:11.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:11.356 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:12.194 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:13.196 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:13.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:13.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:38:13.197 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:38:13.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:13.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:13.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:14.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:14.683 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:14.683 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:38:14.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:38:14.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:14.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:15.730 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:16.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:16.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:16.732 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:38:16.733 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:38:16.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:16.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:17.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:18.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:18.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:18.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:38:18.570 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:38:18.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:18.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:19.067 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:20.069 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:20.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:20.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:38:20.070 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:38:20.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:20.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:21.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:22.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:22.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:22.377 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:38:22.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:38:22.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:22.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:38:23.175 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/255777,v1:127.0.0.1:6811/255777] [v2:127.0.0.1:6812/255777,v1:127.0.0.1:6813/255777] exists,up 481520d0-06bc-43a8-9997-f1d9ad1b0624 2024-09-18T11:38:23.178 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:38:23.179 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:38:23.180 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:38:23.181 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:38:23.181 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:38:23.181 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:38:23.182 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:38:23.182 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:38:23.183 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:38:23.184 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:38:23.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47f33ac1-7902-4a94-bc76-7f105acfc96e 2024-09-18T11:38:23.184 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 47f33ac1-7902-4a94-bc76-7f105acfc96e' 2024-09-18T11:38:23.185 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 47f33ac1-7902-4a94-bc76-7f105acfc96e 2024-09-18T11:38:23.185 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:38:23.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvu+pmbGJADBAAmnNMQZN8QHti5SVz/V0Q3g== 2024-09-18T11:38:23.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvu+pmbGJADBAAmnNMQZN8QHti5SVz/V0Q3g=="}' 2024-09-18T11:38:23.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47f33ac1-7902-4a94-bc76-7f105acfc96e -i td/test-erasure-eio/2/new.json 2024-09-18T11:38:24.048 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:38:24.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:38:24.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvu+pmbGJADBAAmnNMQZN8QHti5SVz/V0Q3g== --osd-uuid 47f33ac1-7902-4a94-bc76-7f105acfc96e 2024-09-18T11:38:24.095 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.089+0000 7f585a564540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:24.097 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.089+0000 7f585a564540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:24.099 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.093+0000 7f585a564540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:24.100 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.093+0000 7f585a564540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:38:24.100 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.093+0000 7f585a564540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:38:24.100 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.093+0000 7f585a564540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:38:24.170 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:24.165+0000 7f585a564540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:38:30.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:38:30.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:38:30.577 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:38:30.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:38:30.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:38:31.139 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:38:31.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:38:31.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:38:31.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:38:31.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:38:31.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:38:31.168 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:31.161+0000 7f0b214cc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:31.171 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:31.165+0000 7f0b214cc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:31.174 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:31.165+0000 7f0b214cc540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:38:31.472 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:31.465+0000 7f0b214cc540 -1 Falling back to public interface 2024-09-18T11:38:31.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:38:31.552 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:38:31.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:31.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:32.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:33.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:33.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:33.106 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:38:33.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:38:33.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:33.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:33.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:34.452 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:38:34.445+0000 7f0b214cc540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:38:34.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:34.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:34.522 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:38:34.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:38:34.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:34.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:35.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:36.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:36.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:36.228 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:38:36.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:38:36.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:36.229 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:36.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:37.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:37.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:37.724 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:38:37.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:38:37.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:37.725 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:38.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:39.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:39.249 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:38:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:38:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:39.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:40.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:40.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:40.740 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:38:40.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:38:40.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:40.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:41.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:42.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:42.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:42.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:38:42.330 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:38:42.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:42.331 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:42.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:43.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:43.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:38:43.810 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:38:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:43.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:44.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:45.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:45.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:45.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:38:45.286 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:38:45.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:45.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:45.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:46.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:46.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:46.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:38:46.701 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:38:46.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:46.701 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:47.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:48.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:48.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:48.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:38:48.140 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:38:48.140 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:48.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:49.146 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:50.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:50.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:50.149 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:38:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:38:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:50.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:51.088 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:52.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:52.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:52.090 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:38:52.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:38:52.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:52.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:52.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:53.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:53.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:53.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:38:53.696 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:38:53.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:53.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:54.156 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:55.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:55.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:55.158 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:38:55.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:38:55.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:55.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:55.966 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:56.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:56.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:56.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:38:56.969 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:38:56.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:56.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:57.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:38:58.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:38:58.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:38:58.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:38:58.764 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:38:58.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:38:58.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:38:59.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:00.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:00.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:00.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:39:00.577 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:39:00.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:00.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:39:01.075 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/257578,v1:127.0.0.1:6819/257578] [v2:127.0.0.1:6820/257578,v1:127.0.0.1:6821/257578] exists,up 47f33ac1-7902-4a94-bc76-7f105acfc96e 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:39:01.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:39:01.079 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:01.080 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:39:01.082 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:39:01.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:39:01.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:39:01.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:39:01.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:39:01.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:39:01.083 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:39:01.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9209d0d-a8a7-4d33-900d-7a636f6a0c3f 2024-09-18T11:39:01.084 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 c9209d0d-a8a7-4d33-900d-7a636f6a0c3f 2024-09-18T11:39:01.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c9209d0d-a8a7-4d33-900d-7a636f6a0c3f' 2024-09-18T11:39:01.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:39:01.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVu+pm9b4cBhAAa3lipyWpIj++CgGs4OGg0w== 2024-09-18T11:39:01.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVu+pm9b4cBhAAa3lipyWpIj++CgGs4OGg0w=="}' 2024-09-18T11:39:01.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9209d0d-a8a7-4d33-900d-7a636f6a0c3f -i td/test-erasure-eio/3/new.json 2024-09-18T11:39:02.372 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:39:02.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:39:02.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVu+pm9b4cBhAAa3lipyWpIj++CgGs4OGg0w== --osd-uuid c9209d0d-a8a7-4d33-900d-7a636f6a0c3f 2024-09-18T11:39:02.411 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.405+0000 7f5776697540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:02.415 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.409+0000 7f5776697540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:02.416 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.409+0000 7f5776697540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:02.417 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.409+0000 7f5776697540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:39:02.418 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.409+0000 7f5776697540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:39:02.418 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.409+0000 7f5776697540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:39:02.675 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:02.669+0000 7f5776697540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:39:09.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:39:09.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:39:09.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:39:09.285 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:39:09.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:39:09.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:39:09.818 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:39:09.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:39:09.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:39:09.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:39:09.819 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:39:09.849 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:09.841+0000 7f99def27540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:09.851 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:09.845+0000 7f99def27540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:09.853 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:09.845+0000 7f99def27540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:10.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:39:10.209 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:39:10.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:10.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:10.424 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:10.417+0000 7f99def27540 -1 Falling back to public interface 2024-09-18T11:39:10.590 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:11.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:11.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:11.592 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:39:11.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:39:11.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:11.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:12.025 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:13.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:13.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:13.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:39:13.028 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:39:13.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:13.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:13.248 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:13.241+0000 7f99def27540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:39:13.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:14.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:14.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:14.471 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:39:14.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:39:14.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:14.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:15.042 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:16.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:39:16.044 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:39:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:16.045 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:16.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:17.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:17.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:17.564 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:39:17.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:39:17.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:17.565 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:18.080 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:19.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:19.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:19.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:39:19.083 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:39:19.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:19.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:19.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:20.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:20.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:20.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:39:20.582 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:39:20.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:20.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:21.141 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:22.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:22.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:22.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:39:22.143 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:39:22.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:22.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:22.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:39:23.577 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:39:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:23.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:24.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:25.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:25.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:25.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:39:25.062 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:39:25.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:25.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:25.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:26.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:26.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:26.583 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:39:26.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:39:26.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:26.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:27.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:28.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:28.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:28.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:39:28.073 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:39:28.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:28.073 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:28.864 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:29.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:29.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:29.867 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:39:29.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:39:29.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:29.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:31.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:32.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:32.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:32.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:39:32.008 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:39:32.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:32.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:32.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:33.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:33.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:33.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:39:33.745 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:39:33.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:33.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:34.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:35.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:35.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:35.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:39:35.613 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:39:35.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:35.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:36.185 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:37.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:37.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:37.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:39:37.188 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:39:37.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:37.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:37.913 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:38.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:38.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:38.916 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:39:38.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:39:38.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:38.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:39:39.521 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/259379,v1:127.0.0.1:6827/259379] [v2:127.0.0.1:6828/259379,v1:127.0.0.1:6829/259379] exists,up c9209d0d-a8a7-4d33-900d-7a636f6a0c3f 2024-09-18T11:39:39.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:39:39.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:39:39.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:39:39.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:39:39.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:39:39.523 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:39:39.524 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:39:39.524 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:39.524 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:39:39.524 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:39:39.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:39:39.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:39:39.605 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:39:39.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:39:39.620 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:37:16.531+0000 7f3032f00540 0 load: jerasure load: lrc 2024-09-18T11:39:39.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:285: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-09-18T11:39:39.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:286: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:39:39.622 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:39:40.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:39:40.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:39:41.626 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:39:41.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:39:42.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:39:42.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:39:42.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:39:42.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:39:42.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:39:42.641 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:39:42.641 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:39:42.642 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:39:42.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:39:42.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:39:42.642 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:39:42.867 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:39:42.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:39:42.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:39:42.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:39:42.868 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:39:43.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:39:43.277 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:39:43.277 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:39:43.278 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:39:43.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:39:43.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:43.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:39:43.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803809 2024-09-18T11:39:43.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803809 2024-09-18T11:39:43.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809' 2024-09-18T11:39:43.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:43.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:39:43.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574876 2024-09-18T11:39:43.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574876 2024-09-18T11:39:43.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-55834574876' 2024-09-18T11:39:43.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:43.599 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:39:43.741 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345940 2024-09-18T11:39:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345940 2024-09-18T11:39:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-55834574876 2-85899345940' 2024-09-18T11:39:43.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:43.742 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:39:43.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018896 2024-09-18T11:39:43.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018896 2024-09-18T11:39:43.885 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803809 1-55834574876 2-85899345940 3-128849018896' 2024-09-18T11:39:43.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:43.886 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803809 2024-09-18T11:39:43.886 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:43.889 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:39:43.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803809 2024-09-18T11:39:43.890 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:43.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803809 2024-09-18T11:39:43.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-09-18T11:39:43.892 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803809 2024-09-18T11:39:43.892 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:39:44.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803807 -lt 25769803809 2024-09-18T11:39:44.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:39:45.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:39:45.385 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:39:45.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803809 2024-09-18T11:39:45.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:45.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574876 2024-09-18T11:39:45.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:45.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:39:45.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574876 2024-09-18T11:39:45.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:45.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574876 2024-09-18T11:39:45.809 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574876 2024-09-18T11:39:45.810 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-09-18T11:39:45.810 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:39:46.231 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574876 -lt 55834574876 2024-09-18T11:39:46.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:46.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:46.233 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345940 2024-09-18T11:39:46.234 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:39:46.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345940 2024-09-18T11:39:46.235 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:46.237 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345940 2024-09-18T11:39:46.238 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 85899345940 2024-09-18T11:39:46.238 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345940' 2024-09-18T11:39:46.238 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:39:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345940 -lt 85899345940 2024-09-18T11:39:46.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:46.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:46.671 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018896 2024-09-18T11:39:46.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:39:46.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018896 2024-09-18T11:39:46.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:46.676 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018896 2024-09-18T11:39:46.677 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018896' 2024-09-18T11:39:46.677 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 128849018896 2024-09-18T11:39:46.677 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:39:47.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018896 -lt 128849018896 2024-09-18T11:39:47.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:39:47.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:39:47.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:39:47.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:39:47.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:39:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:39:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:39:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:39:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:39:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:39:47.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:39:48.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:39:48.067 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:39:48.067 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:39:48.068 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:39:48.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:39:48.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-18T11:39:48.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:39:48.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-18T11:39:48.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:39:48.571 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:39:48.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:39:48.672 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:39:48.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:39:48.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:39:48.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:39:48.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:39:48.673 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:39:48.674 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:39:49.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:39:49.108 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:39:49.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:39:49.109 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:39:49.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:39:49.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:39:49.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:39:49.655 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_missing: local shard_id=1 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:289: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-79554-1 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-79554-1 2024-09-18T11:39:49.656 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-79554-1 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:39:49.657 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-79554-1 td/test-erasure-eio/ORIGINAL 2024-09-18T11:39:49.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-79554-1 td/test-erasure-eio 1 2024-09-18T11:39:49.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-18T11:39:49.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-79554-1 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-79554-1 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:39:49.807 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-79554-1 2024-09-18T11:39:49.808 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-79554-1 2024-09-18T11:39:49.808 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:39:50.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:39:50.206 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:39:50.206 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:39:50.206 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-79554-1 remove 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-79554-1 remove 2024-09-18T11:39:50.208 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:39:50.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:39:50.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:39:50.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-79554-1 remove 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-09-18T11:39:50.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-79554-1 remove 2024-09-18T11:39:52.466 INFO:tasks.workunit.client.0.smithi174.stdout:remove 1#2:f5c1880b:::obj-remove-79554-1:head# 2024-09-18T11:39:53.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-09-18T11:39:53.000 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:39:53.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:39:53.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:39:53.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:39:53.003 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:39:53.003 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:39:53.003 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:39:53.003 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:39:53.003 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:39:53.004 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:39:53.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:39:53.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:39:53.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:39:53.005 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-18T11:39:53.005 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.1 2024-09-18T11:39:53.006 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:39:53.007 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-09-18T11:39:53.008 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-18T11:39:53.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:39:53.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:39:53.009 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:39:53.036 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:53.032+0000 7f3e6264e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:53.042 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:53.036+0000 7f3e6264e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:53.043 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:53.040+0000 7f3e6264e540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:39:53.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-18T11:39:53.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:39:53.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:39:53.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:39:53.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:39:53.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:53.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:39:53.403 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:39:53.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:53.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:53.592 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:53.588+0000 7f3e6264e540 -1 Falling back to public interface 2024-09-18T11:39:53.866 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:54.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:54.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:54.868 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:39:54.869 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:39:54.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:54.869 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:55.314 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:39:56.297 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:39:56.292+0000 7f3e6264e540 -1 osd.1 38 log_to_monitors true 2024-09-18T11:39:56.316 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:39:56.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:39:56.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:39:56.317 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:39:56.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:39:56.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:osd.1 up in weight 1 up_from 42 up_thru 21 down_at 39 last_clean_interval [13,38) [v2:127.0.0.1:6810/261350,v1:127.0.0.1:6811/261350] [v2:127.0.0.1:6812/261350,v1:127.0.0.1:6813/261350] exists,up 481520d0-06bc-43a8-9997-f1d9ad1b0624 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:39:56.952 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:39:56.953 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:39:56.953 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:39:56.953 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:39:56.953 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:39:56.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:39:56.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:39:56.954 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:39:57.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:39:57.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:39:57.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:39:57.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:39:57.168 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:39:57.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:39:57.789 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:39:57.789 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:39:57.789 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:39:57.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:39:57.789 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:57.790 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:39:58.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803814 2024-09-18T11:39:58.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803814 2024-09-18T11:39:58.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814' 2024-09-18T11:39:58.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:58.002 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:39:58.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626435 2024-09-18T11:39:58.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626435 2024-09-18T11:39:58.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-180388626435' 2024-09-18T11:39:58.147 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:58.147 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:39:58.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345945 2024-09-18T11:39:58.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345945 2024-09-18T11:39:58.388 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-180388626435 2-85899345945' 2024-09-18T11:39:58.389 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:39:58.389 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:39:58.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018900 2024-09-18T11:39:58.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018900 2024-09-18T11:39:58.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-180388626435 2-85899345945 3-128849018900' 2024-09-18T11:39:58.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:39:58.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803814 2024-09-18T11:39:58.536 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:39:58.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:39:58.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803814 2024-09-18T11:39:58.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:39:58.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803814 2024-09-18T11:39:58.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-18T11:39:58.541 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803814 2024-09-18T11:39:58.542 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:39:59.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803812 -lt 25769803814 2024-09-18T11:39:59.014 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:40:00.015 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:40:00.016 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:40:00.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803815 -lt 25769803814 2024-09-18T11:40:00.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:00.428 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:00.428 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-180388626435 2024-09-18T11:40:00.428 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:40:00.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-180388626435 2024-09-18T11:40:00.429 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:00.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626435 2024-09-18T11:40:00.432 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 180388626435' 2024-09-18T11:40:00.432 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 180388626435 2024-09-18T11:40:00.433 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:40:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626435 -lt 180388626435 2024-09-18T11:40:00.847 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:00.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345945 2024-09-18T11:40:00.848 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:00.850 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:40:00.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:00.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345945 2024-09-18T11:40:00.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345945 2024-09-18T11:40:00.852 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345945' 2024-09-18T11:40:00.852 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 85899345945 2024-09-18T11:40:00.853 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:40:01.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345946 -lt 85899345945 2024-09-18T11:40:01.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:01.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018900 2024-09-18T11:40:01.279 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:01.281 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:40:01.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:01.282 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018900 2024-09-18T11:40:01.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018900 2024-09-18T11:40:01.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018900' 2024-09-18T11:40:01.283 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018900 2024-09-18T11:40:01.284 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:40:01.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018901 -lt 128849018900 2024-09-18T11:40:01.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:40:01.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:40:01.720 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:40:02.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:40:02.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:40:02.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:40:02.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:40:02.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:40:02.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:40:02.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:40:02.227 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:40:02.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:40:02.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:40:02.666 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:40:02.667 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:40:03.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:40:03.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:40:03.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:40:03.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-79554-1 2024-09-18T11:40:03.172 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:40:03.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:40:03.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-79554-1 2024-09-18T11:40:03.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:40:03.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:40:03.173 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-79554-1 td/test-erasure-eio/COPY 2024-09-18T11:40:03.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:40:03.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:40:03.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-09-18T11:40:03.255 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-79554-1 td/test-erasure-eio 2 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-09-18T11:40:03.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-79554-1 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-09-18T11:40:03.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:40:03.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-79554-1 2024-09-18T11:40:03.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:40:03.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-79554-1 2024-09-18T11:40:03.259 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-79554-1 2024-09-18T11:40:03.259 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:40:03.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:40:03.648 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:40:03.648 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:40:03.648 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-79554-1 remove 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-79554-1 remove 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-18T11:40:03.650 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-18T11:40:03.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-18T11:40:03.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:40:03.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:40:03.651 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:40:03.651 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:40:03.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:40:03.652 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-79554-1 remove 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-09-18T11:40:03.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-79554-1 remove 2024-09-18T11:40:06.211 INFO:tasks.workunit.client.0.smithi174.stdout:remove 2#2:f5c1880b:::obj-remove-79554-1:head# 2024-09-18T11:40:06.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:40:06.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:40:06.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:40:06.750 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:40:06.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:06.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:06.751 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:06.751 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:06.751 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:06.751 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:06.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:06.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:40:06.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:40:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:start osd.2 2024-09-18T11:40:06.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:40:06.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:40:06.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:40:06.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:40:06.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:40:06.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:40:06.784 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:06.780+0000 7f7738a94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:06.787 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:06.784+0000 7f7738a94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:06.791 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:06.784+0000 7f7738a94540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:40:07.166 INFO:tasks.workunit.client.0.smithi174.stderr:0 2024-09-18T11:40:07.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:07.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:40:07.348 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:07.344+0000 7f7738a94540 -1 Falling back to public interface 2024-09-18T11:40:07.607 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:08.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:08.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:08.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:40:08.611 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:40:08.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:08.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:40:09.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:09.943 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:09.940+0000 7f7738a94540 -1 osd.2 44 log_to_monitors true 2024-09-18T11:40:10.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:10.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:10.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:40:10.036 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:40:10.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:10.037 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:40:10.481 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:11.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:11.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:11.483 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:40:11.484 INFO:tasks.workunit.client.0.smithi174.stderr:3 2024-09-18T11:40:11.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:11.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:40:11.947 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 up in weight 1 up_from 49 up_thru 39 down_at 45 last_clean_interval [20,44) [v2:127.0.0.1:6818/262799,v1:127.0.0.1:6819/262799] [v2:127.0.0.1:6820/262799,v1:127.0.0.1:6821/262799] exists,up 47f33ac1-7902-4a94-bc76-7f105acfc96e 2024-09-18T11:40:11.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:40:11.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:40:11.947 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:40:11.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-18T11:40:11.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:40:11.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:40:11.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:40:11.948 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:40:11.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:40:11.949 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:40:11.949 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:40:11.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:40:11.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:40:11.950 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:40:12.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:40:12.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:40:12.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:40:12.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:40:12.168 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:40:12.581 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:40:12.581 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:40:12.581 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:40:12.582 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:40:12.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:40:12.582 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:40:12.582 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:40:12.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803819 2024-09-18T11:40:12.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803819 2024-09-18T11:40:12.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819' 2024-09-18T11:40:12.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:40:12.753 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:40:12.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=180388626440 2024-09-18T11:40:12.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 180388626440 2024-09-18T11:40:12.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-180388626440' 2024-09-18T11:40:12.933 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:40:12.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:40:13.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397507 2024-09-18T11:40:13.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397507 2024-09-18T11:40:13.077 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-180388626440 2-210453397507' 2024-09-18T11:40:13.078 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:40:13.078 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:40:13.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018905 2024-09-18T11:40:13.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018905 2024-09-18T11:40:13.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-180388626440 2-210453397507 3-128849018905' 2024-09-18T11:40:13.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:13.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803819 2024-09-18T11:40:13.218 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:13.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:40:13.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803819 2024-09-18T11:40:13.221 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:13.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803819 2024-09-18T11:40:13.223 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-09-18T11:40:13.223 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.0 seq 25769803819 2024-09-18T11:40:13.223 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:40:13.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803819 -lt 25769803819 2024-09-18T11:40:13.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:13.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-180388626440 2024-09-18T11:40:13.636 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:13.638 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:40:13.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-180388626440 2024-09-18T11:40:13.640 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:13.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=180388626440 2024-09-18T11:40:13.642 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 180388626440' 2024-09-18T11:40:13.642 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.1 seq 180388626440 2024-09-18T11:40:13.643 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:40:14.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626438 -lt 180388626440 2024-09-18T11:40:14.070 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:40:15.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:40:15.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:40:15.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 180388626440 -lt 180388626440 2024-09-18T11:40:15.531 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:15.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-210453397507 2024-09-18T11:40:15.532 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:15.534 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:40:15.535 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-210453397507 2024-09-18T11:40:15.535 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:15.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397507 2024-09-18T11:40:15.537 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 210453397507' 2024-09-18T11:40:15.537 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.2 seq 210453397507 2024-09-18T11:40:15.538 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:40:15.985 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397507 -lt 210453397507 2024-09-18T11:40:15.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:40:15.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:40:15.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018905 2024-09-18T11:40:15.988 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:40:15.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018905 2024-09-18T11:40:15.989 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:40:15.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018905 2024-09-18T11:40:15.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018905' 2024-09-18T11:40:15.990 INFO:tasks.workunit.client.0.smithi174.stderr:waiting osd.3 seq 128849018905 2024-09-18T11:40:15.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:40:16.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018906 -lt 128849018905 2024-09-18T11:40:16.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:40:16.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:40:16.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:40:16.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:40:16.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:40:16.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:40:16.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:40:16.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:40:16.980 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:40:16.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:40:16.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:40:17.386 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:40:17.386 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:40:17.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:40:17.387 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:40:17.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:40:17.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-79554-1 fail 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-79554-1 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-09-18T11:40:17.876 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-remove-79554-1 td/test-erasure-eio/COPY 2024-09-18T11:40:17.937 INFO:tasks.workunit.client.0.smithi174.stderr:error getting pool-jerasure/obj-remove-79554-1: (5) Input/output error 2024-09-18T11:40:17.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-09-18T11:40:17.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-09-18T11:40:17.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:40:17.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:40:17.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:40:18.625 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:40:18.635 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:40:19.269 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:40:19.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:40:19.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:40:19.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:40:19.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:40:19.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:40:19.278 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:40:19.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:40:19.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:40:19.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:40:19.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:40:19.397 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:40:19.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:40:19.399 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:40:19.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:40:19.402 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:40:19.402 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:40:19.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:19.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:40:19.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:40:19.406 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:19.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:40:19.409 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:40:19.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:40:19.432 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:40:19.433 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:19.433 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:19.433 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:40:19.435 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:40:19.436 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:40:19.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:40:19.437 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:40:19.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:40:19.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:40:19.437 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:40:19.441 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:40:19.442 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:40:19.444 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:40:19.444 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:40:19.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:40:19.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:40:19.446 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:40:19.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:19.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:40:19.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:40:19.449 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:40:19.450 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:40:19.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:40:19.452 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:40:19.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:40:19.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:19.454 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:19.454 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:40:19.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:40:19.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:40:19.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:40:19.458 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:40:19.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:19.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:19.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:40:19.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:40:19.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:40:19.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:40:19.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:40:19.888 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:40:19.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:19.889 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:19.889 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:19.889 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:19.889 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:19.890 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:19.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:40:20.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:40:20.171 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:40:20.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:40:20.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:40:20.172 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:40:20.174 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:40:20.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:40:20.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:40:20.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:40:20.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:40:20.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:40:20.267 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:40:20.267 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:40:20.268 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:40:20.268 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:40:20.268 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:40:20.268 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:20.268 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:20.268 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:40:20.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:40:20.269 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:40:20.350 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:40:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:40:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:40:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:40:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:40:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:40:20.351 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:40:21.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:40:21.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:21.272 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:21.273 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:21.273 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:21.273 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:21.273 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:21.274 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:40:21.275 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:40:21.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:40:21.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:40:21.695 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:40:21.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:40:22.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:40:22.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:40:22.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:40:22.707 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:40:22.707 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:22.707 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:22.707 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:40:22.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:40:22.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:40:22.785 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:40:22.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:40:22.795 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:40:19.928+0000 7efd1a4b6a00 0 load: jerasure load: lrc 2024-09-18T11:40:22.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-09-18T11:40:22.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-09-18T11:40:22.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-09-18T11:40:22.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:333: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-09-18T11:40:22.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:40:22.796 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:40:22.797 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:40:22.798 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:40:22.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:40:22.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:40:22.799 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:40:22.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:40:22.801 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:40:22.801 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:22.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:22.802 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:22.802 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:22.802 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:22.802 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:40:22.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:40:22.804 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:40:22.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6374c5c5-5d56-49c8-907a-097b337a7892 2024-09-18T11:40:22.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6374c5c5-5d56-49c8-907a-097b337a7892' 2024-09-18T11:40:22.805 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 6374c5c5-5d56-49c8-907a-097b337a7892 2024-09-18T11:40:22.806 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:40:22.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmvOpmgKgLMRAAwDfmtaz/TVQP+t4twq/mDw== 2024-09-18T11:40:22.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmvOpmgKgLMRAAwDfmtaz/TVQP+t4twq/mDw=="}' 2024-09-18T11:40:22.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6374c5c5-5d56-49c8-907a-097b337a7892 -i td/test-erasure-eio/0/new.json 2024-09-18T11:40:23.173 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:40:23.188 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:40:23.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmvOpmgKgLMRAAwDfmtaz/TVQP+t4twq/mDw== --osd-uuid 6374c5c5-5d56-49c8-907a-097b337a7892 2024-09-18T11:40:23.221 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.216+0000 7f40ba3ed540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:23.223 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.220+0000 7f40ba3ed540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:23.225 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.220+0000 7f40ba3ed540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:23.225 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.220+0000 7f40ba3ed540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:40:23.226 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.220+0000 7f40ba3ed540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:40:23.226 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.220+0000 7f40ba3ed540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:40:23.314 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:23.308+0000 7f40ba3ed540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:40:29.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:40:29.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:40:29.165 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:40:29.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:40:29.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:40:29.699 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:40:29.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:40:29.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:40:29.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:40:29.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:40:29.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:40:29.728 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:29.724+0000 7f4105656540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:29.731 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:29.728+0000 7f4105656540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:29.732 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:29.728+0000 7f4105656540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:30.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:40:30.139 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:40:30.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:30.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:30.292 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:30.288+0000 7f4105656540 -1 Falling back to public interface 2024-09-18T11:40:30.543 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:31.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:31.546 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:40:31.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:40:31.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:31.546 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:31.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:32.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:32.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:32.934 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:40:32.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:40:32.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:32.935 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:33.066 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:33.064+0000 7f4105656540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:40:33.347 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:34.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:34.348 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:34.349 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:40:34.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:40:34.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:34.349 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:34.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:35.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:35.818 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:35.819 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:40:35.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:40:35.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:35.820 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:36.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:37.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:37.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:37.255 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:40:37.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:40:37.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:37.255 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:37.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:38.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:38.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:38.807 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:40:38.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:40:38.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:38.807 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:39.249 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:40.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:40.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:40.251 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:40:40.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:40:40.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:40.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:40.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:41.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:41.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:41.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:40:41.831 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:40:41.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:41.831 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:42.244 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:40:43.247 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:40:43.247 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:43.248 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:43.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:44.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:44.827 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:44.827 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:40:44.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:40:44.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:44.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:45.217 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:46.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:46.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:46.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:40:46.219 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:40:46.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:46.220 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:46.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:47.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:47.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:47.803 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:40:47.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:40:47.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:47.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:48.519 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:49.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:49.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:49.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:40:49.522 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:40:49.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:49.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:50.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:51.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:51.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:51.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:40:51.754 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:40:51.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:51.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:52.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:53.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:53.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:53.905 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:40:53.906 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:40:53.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:53.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:55.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:56.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:56.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:56.302 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:40:56.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:40:56.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:56.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:56.721 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:40:57.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:40:57.723 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:40:57.723 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:40:57.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:40:57.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:40:57.724 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:40:58.250 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/265070,v1:127.0.0.1:6803/265070] [v2:127.0.0.1:6804/265070,v1:127.0.0.1:6805/265070] exists,up 6374c5c5-5d56-49c8-907a-097b337a7892 2024-09-18T11:40:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:40:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:40:58.252 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:40:58.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:40:58.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:40:58.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:40:58.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:40:58.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:40:58.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:40:58.254 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:40:58.256 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:40:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:40:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:40:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:40:58.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:40:58.257 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:40:58.258 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14275d56-a023-4984-8f40-bf3f97a57152 2024-09-18T11:40:58.258 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 14275d56-a023-4984-8f40-bf3f97a57152 2024-09-18T11:40:58.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 14275d56-a023-4984-8f40-bf3f97a57152' 2024-09-18T11:40:58.259 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:40:58.278 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKvOpmnNh7EBAAgM34lcngSpSpa58tmnwsPw== 2024-09-18T11:40:58.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKvOpmnNh7EBAAgM34lcngSpSpa58tmnwsPw=="}' 2024-09-18T11:40:58.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14275d56-a023-4984-8f40-bf3f97a57152 -i td/test-erasure-eio/1/new.json 2024-09-18T11:40:58.797 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:40:58.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:40:58.811 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKvOpmnNh7EBAAgM34lcngSpSpa58tmnwsPw== --osd-uuid 14275d56-a023-4984-8f40-bf3f97a57152 2024-09-18T11:40:58.835 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.831+0000 7fb6ec838540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:58.838 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.835+0000 7fb6ec838540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:58.839 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.835+0000 7fb6ec838540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:40:58.839 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.835+0000 7fb6ec838540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:40:58.840 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.835+0000 7fb6ec838540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:40:58.840 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.835+0000 7fb6ec838540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:40:58.932 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:40:58.927+0000 7fb6ec838540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:41:05.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:41:05.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:41:05.221 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:41:05.221 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:41:05.222 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:41:05.756 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:41:05.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:41:05.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:41:05.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:41:05.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:41:05.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:41:05.787 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:05.783+0000 7f8733efd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:05.790 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:05.787+0000 7f8733efd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:05.795 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:05.787+0000 7f8733efd540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:06.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:41:06.167 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:41:06.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:06.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:06.332 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:06.327+0000 7f8733efd540 -1 Falling back to public interface 2024-09-18T11:41:06.593 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:07.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:07.594 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:07.595 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:41:07.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:41:07.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:07.595 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:07.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:09.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:09.001 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:09.001 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:41:09.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:41:09.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:09.002 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:09.115 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:09.111+0000 7f8733efd540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:41:09.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:10.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:10.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:10.448 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:41:10.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:41:10.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:10.449 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:10.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:11.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:11.971 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:11.971 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:41:11.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:41:11.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:11.972 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:12.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:13.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:13.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:13.459 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:41:13.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:41:13.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:13.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:13.875 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:14.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:14.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:14.878 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:41:14.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:41:14.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:14.878 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:15.306 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:16.307 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:16.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:16.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:41:16.308 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:41:16.308 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:16.309 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:16.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:17.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:17.748 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:17.749 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:41:17.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:41:17.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:17.749 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:18.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:19.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:19.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:19.286 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:41:19.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:41:19.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:19.287 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:19.806 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:20.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:20.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:20.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:41:20.809 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:41:20.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:20.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:21.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:22.296 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:22.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:22.297 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:41:22.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:41:22.297 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:22.298 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:22.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:23.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:23.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:23.792 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:41:23.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:41:23.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:23.793 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:24.687 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:25.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:25.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:25.690 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:41:25.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:41:25.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:25.691 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:26.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:27.761 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:41:27.762 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:41:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:27.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:28.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:29.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:29.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:29.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:41:29.892 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:41:29.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:29.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:30.473 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:31.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:31.475 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:31.476 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:41:31.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:41:31.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:31.476 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:32.533 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:33.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:33.535 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:33.536 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:41:33.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:41:33.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:33.536 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:41:34.032 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/266845,v1:127.0.0.1:6811/266845] [v2:127.0.0.1:6812/266845,v1:127.0.0.1:6813/266845] exists,up 14275d56-a023-4984-8f40-bf3f97a57152 2024-09-18T11:41:34.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:41:34.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:41:34.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:41:34.036 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:41:34.038 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:41:34.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:41:34.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:41:34.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:41:34.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:41:34.039 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:41:34.039 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:41:34.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af5f3a42-ad46-42b6-b32b-77a5283e5652 2024-09-18T11:41:34.040 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 af5f3a42-ad46-42b6-b32b-77a5283e5652' 2024-09-18T11:41:34.040 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 af5f3a42-ad46-42b6-b32b-77a5283e5652 2024-09-18T11:41:34.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:41:34.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuvOpm8XFgAxAAfFO+x32hBulWLYXSs4YeEg== 2024-09-18T11:41:34.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuvOpm8XFgAxAAfFO+x32hBulWLYXSs4YeEg=="}' 2024-09-18T11:41:34.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af5f3a42-ad46-42b6-b32b-77a5283e5652 -i td/test-erasure-eio/2/new.json 2024-09-18T11:41:35.163 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:41:35.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:41:35.176 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBuvOpm8XFgAxAAfFO+x32hBulWLYXSs4YeEg== --osd-uuid af5f3a42-ad46-42b6-b32b-77a5283e5652 2024-09-18T11:41:35.207 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.203+0000 7fc8f63cf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:35.333 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.327+0000 7fc8f63cf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:35.335 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.331+0000 7fc8f63cf540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:35.336 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.331+0000 7fc8f63cf540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:41:35.336 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.331+0000 7fc8f63cf540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:41:35.336 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.331+0000 7fc8f63cf540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:41:35.452 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:35.447+0000 7fc8f63cf540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:41:41.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:41:41.871 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:41:41.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:41:41.873 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:41:41.873 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:41:42.405 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:41:42.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:41:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:41:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:41:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:41:42.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:41:42.436 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:42.431+0000 7f877659a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:42.439 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:42.435+0000 7f877659a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:42.442 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:42.439+0000 7f877659a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:41:42.724 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:42.719+0000 7f877659a540 -1 Falling back to public interface 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:42.886 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:41:42.887 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:41:42.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:42.887 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:43.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:44.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:44.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:44.266 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:41:44.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:41:44.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:44.267 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:44.682 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:45.610 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:41:45.603+0000 7f877659a540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:41:45.684 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:45.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:45.703 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:41:45.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:41:45.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:45.704 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:46.269 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:47.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:47.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:47.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:41:47.272 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:41:47.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:47.272 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:47.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:48.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:48.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:48.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:41:48.909 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:41:48.909 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:48.910 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:49.382 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:50.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:50.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:50.384 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:41:50.384 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:41:50.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:50.385 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:50.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:51.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:51.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:51.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:41:51.816 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:41:51.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:51.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:52.210 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:53.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:53.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:53.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:41:53.212 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:41:53.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:53.213 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:53.823 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:54.825 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:54.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:54.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:41:54.826 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:41:54.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:54.826 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:55.239 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:56.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:56.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:56.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:41:56.242 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:41:56.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:56.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:56.736 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:57.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:57.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:57.738 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:41:57.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:41:57.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:57.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:41:58.167 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:41:59.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:41:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:41:59.170 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:41:59.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:41:59.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:41:59.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:00.166 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:01.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:01.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:01.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:42:01.168 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:42:01.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:01.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:01.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:02.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:02.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:02.794 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:42:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:42:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:02.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:03.812 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:04.815 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:42:04.815 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:42:04.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:04.816 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:05.520 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:06.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:06.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:06.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:42:06.522 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:42:06.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:06.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:07.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:08.227 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:08.227 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:42:08.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:42:08.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:08.228 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:09.063 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:10.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:10.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:10.065 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:42:10.066 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:42:10.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:10.066 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:42:10.536 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/268620,v1:127.0.0.1:6819/268620] [v2:127.0.0.1:6820/268620,v1:127.0.0.1:6821/268620] exists,up af5f3a42-ad46-42b6-b32b-77a5283e5652 2024-09-18T11:42:10.538 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:42:10.539 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:42:10.540 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:42:10.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:42:10.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:10.541 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:42:10.541 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:42:10.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:42:10.541 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:42:10.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:42:10.543 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:42:10.544 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 b2cd45a0-57a2-4114-b189-3ac7caac7f13 2024-09-18T11:42:10.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2cd45a0-57a2-4114-b189-3ac7caac7f13 2024-09-18T11:42:10.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b2cd45a0-57a2-4114-b189-3ac7caac7f13' 2024-09-18T11:42:10.545 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:42:10.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSvOpmb/W3IRAAcaugCfGFYnAba1OFPfwxbg== 2024-09-18T11:42:10.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSvOpmb/W3IRAAcaugCfGFYnAba1OFPfwxbg=="}' 2024-09-18T11:42:10.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2cd45a0-57a2-4114-b189-3ac7caac7f13 -i td/test-erasure-eio/3/new.json 2024-09-18T11:42:11.749 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:42:11.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:42:11.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSvOpmb/W3IRAAcaugCfGFYnAba1OFPfwxbg== --osd-uuid b2cd45a0-57a2-4114-b189-3ac7caac7f13 2024-09-18T11:42:11.796 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.790+0000 7f04a104f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:11.799 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.794+0000 7f04a104f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:11.800 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.794+0000 7f04a104f540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:11.801 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.794+0000 7f04a104f540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:42:11.801 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.794+0000 7f04a104f540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:42:11.801 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.794+0000 7f04a104f540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:42:11.855 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:11.850+0000 7f04a104f540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:42:18.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:42:18.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:42:18.417 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:42:18.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:42:18.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:42:18.959 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:42:18.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:42:18.959 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:42:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:42:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:42:18.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:42:18.988 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:18.982+0000 7f1978663540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:19.030 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:19.026+0000 7f1978663540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:19.031 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:19.026+0000 7f1978663540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:42:19.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:42:19.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:42:19.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:42:19.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:42:19.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:42:19.352 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:19.353 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:42:19.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:42:19.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:19.353 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:19.613 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:19.606+0000 7f1978663540 -1 Falling back to public interface 2024-09-18T11:42:19.792 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:20.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:20.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:20.794 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:42:20.794 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:42:20.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:20.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:21.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:22.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:22.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:22.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:42:22.211 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:42:22.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:22.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:22.370 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:42:22.366+0000 7f1978663540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:42:22.690 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:23.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:23.692 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:23.692 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:42:23.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:42:23.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:23.693 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:24.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:25.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:25.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:25.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:42:25.318 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:42:25.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:25.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:25.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:26.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:26.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:26.757 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:42:26.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:42:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:26.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:27.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:28.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:28.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:28.232 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:42:28.232 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:42:28.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:28.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:28.700 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:29.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:29.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:29.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:42:29.703 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:42:29.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:29.703 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:30.209 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:31.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:31.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:31.211 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:42:31.212 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:42:31.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:31.212 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:31.731 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:32.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:32.733 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:32.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:42:32.734 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:42:32.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:32.734 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:33.291 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:34.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:34.293 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:34.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:42:34.294 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:42:34.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:34.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:34.735 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:35.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:35.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:35.737 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:42:35.737 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:42:35.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:35.738 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:36.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:37.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:37.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:37.324 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:42:37.324 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:42:37.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:37.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:38.401 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:39.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:39.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:39.403 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:42:39.404 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:42:39.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:39.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:40.277 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:41.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:41.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:41.279 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:42:41.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:42:41.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:41.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:41.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:42.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:42.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:42.845 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:42:42.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:42:42.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:42.846 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:43.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:44.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:44.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:44.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:42:44.788 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:42:44.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:44.788 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:45.780 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:46.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:46.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:46.783 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:42:46.784 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:42:46.784 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:46.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:47.716 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:42:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:42:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:42:48.719 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:42:48.720 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:42:48.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:42:48.720 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:42:49.174 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/270397,v1:127.0.0.1:6827/270397] [v2:127.0.0.1:6828/270397,v1:127.0.0.1:6829/270397] exists,up b2cd45a0-57a2-4114-b189-3ac7caac7f13 2024-09-18T11:42:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:42:49.174 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:42:49.175 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:42:49.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:42:49.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:42:49.175 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:42:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:42:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:49.176 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:42:49.176 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:42:49.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:42:49.177 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:42:49.265 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:42:49.271 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:42:49.278 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:40:30.300+0000 7f4105656540 0 load: jerasure load: lrc 2024-09-18T11:42:49.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:335: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-09-18T11:42:49.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:42:49.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:42:49.279 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:42:49.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:42:49.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:42:49.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:42:49.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:42:49.280 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:42:50.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:42:50.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:42:52.219 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:42:52.230 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:42:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:42:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:42:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:42:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:42:53.233 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:42:53.234 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:42:53.235 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:42:53.235 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:42:53.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:42:53.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:42:53.236 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:42:53.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:42:53.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:42:53.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:42:53.457 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:42:53.457 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:42:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:42:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:42:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:42:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:42:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:42:53.961 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:42:53.962 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:42:54.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803810 2024-09-18T11:42:54.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803810 2024-09-18T11:42:54.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810' 2024-09-18T11:42:54.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:42:54.159 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:42:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574871 2024-09-18T11:42:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574871 2024-09-18T11:42:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574871' 2024-09-18T11:42:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:42:54.319 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:42:54.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378642 2024-09-18T11:42:54.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378642 2024-09-18T11:42:54.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574871 2-81604378642' 2024-09-18T11:42:54.465 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:42:54.465 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:42:54.609 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084303 2024-09-18T11:42:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084303 2024-09-18T11:42:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803810 1-55834574871 2-81604378642 3-120259084303' 2024-09-18T11:42:54.610 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:42:54.611 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803810 2024-09-18T11:42:54.611 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:42:54.612 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:42:54.613 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803810 2024-09-18T11:42:54.614 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:42:54.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803810 2024-09-18T11:42:54.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-09-18T11:42:54.616 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803810 2024-09-18T11:42:54.617 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:42:55.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803810 -lt 25769803810 2024-09-18T11:42:55.026 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:42:55.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574871 2024-09-18T11:42:55.027 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:42:55.029 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:42:55.030 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574871 2024-09-18T11:42:55.031 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:42:55.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574871 2024-09-18T11:42:55.033 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574871 2024-09-18T11:42:55.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574871' 2024-09-18T11:42:55.034 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:42:55.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574871 -lt 55834574871 2024-09-18T11:42:55.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:42:55.584 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378642 2024-09-18T11:42:55.584 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:42:55.587 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:42:55.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:42:55.588 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378642 2024-09-18T11:42:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378642 2024-09-18T11:42:55.589 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378642' 2024-09-18T11:42:55.589 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378642 2024-09-18T11:42:55.589 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:42:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378642 -lt 81604378642 2024-09-18T11:42:55.976 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:42:55.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084303 2024-09-18T11:42:55.977 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:42:55.979 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:42:55.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084303 2024-09-18T11:42:55.981 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:42:55.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084303 2024-09-18T11:42:55.983 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084303' 2024-09-18T11:42:55.983 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084303 2024-09-18T11:42:55.983 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:42:56.419 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084301 -lt 120259084303 2024-09-18T11:42:56.420 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:42:57.421 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:42:57.422 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:42:57.838 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084303 -lt 120259084303 2024-09-18T11:42:57.838 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:42:57.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:42:57.839 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:42:58.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:42:58.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:42:58.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:42:58.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:42:58.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:42:58.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:42:58.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:42:58.329 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:42:58.840 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:42:58.841 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:42:58.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:42:58.842 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:338: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-09-18T11:42:59.336 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79554-0 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79554-0 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79554-0 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:42:59.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:42:59.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:42:59.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:42:59.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:42:59.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:42:59.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79554-0 td/test-erasure-eio/ORIGINAL 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79554-0 td/test-erasure-eio 0 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-79554-0 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:42:59.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:42:59.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:42:59.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-18T11:42:59.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:42:59.505 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:42:59.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-79554-0 2024-09-18T11:42:59.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:42:59.506 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-0 2024-09-18T11:42:59.506 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-0 2024-09-18T11:42:59.506 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-18T11:42:59.986 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:42:59.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/3/type 2024-09-18T11:42:59.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:42:59.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-18T11:42:59.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:42:59.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-18T11:42:59.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:42:59.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:42:59.991 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:42:59.992 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-18T11:42:59.992 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:42:59.992 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:42:59.993 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:42:59.993 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:42:59.993 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:42:59.993 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:42:59.994 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:43:00.101 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:43:00.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:43:00.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:43:00.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-18T11:43:00.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-18T11:43:00.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-18T11:43:00.104 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:43:00.104 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:00.104 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:00.104 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.3.asok 2024-09-18T11:43:00.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:43:00.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-79554-0 0 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79554-0 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79554-0 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:43:00.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79554-0 td/test-erasure-eio/COPY 2024-09-18T11:43:00.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:43:00.241 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:43:00.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-18T11:43:00.242 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:43:00.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-79554-0 2024-09-18T11:43:00.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:43:00.243 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-0 2024-09-18T11:43:00.244 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-0 2024-09-18T11:43:00.244 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:43:00.643 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:43:00.644 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:43:00.644 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:43:00.644 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:43:00.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-18T11:43:00.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-18T11:43:00.644 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:43:00.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:43:00.645 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:43:00.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:43:00.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:43:00.646 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:43:00.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:43:00.960 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-18T11:43:01.910 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-18T11:43:01.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-79554-0 2024-09-18T11:43:01.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-18T11:43:01.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:43:01.919 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-0 2024-09-18T11:43:01.920 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-0 2024-09-18T11:43:01.921 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:43:02.532 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:43:02.544 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:43:02.544 INFO:tasks.workunit.client.0.smithi174.stderr:0' 2024-09-18T11:43:02.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T11:43:02.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-18T11:43:03.448 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:43:03.463 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:43:03.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:43:03.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:43:03.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:03.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:43:03.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:43:03.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:43:03.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:43:03.468 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:43:03.468 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:43:03.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:43:03.470 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:43:03.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:43:03.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:43:03.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:43:03.472 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:43:03.495 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:03.490+0000 7f85401d2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:03.520 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:03.514+0000 7f85401d2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:03.521 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:03.514+0000 7f85401d2540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:03.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:43:03.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:43:03.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:03.916 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:43:04.080 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:04.074+0000 7f85401d2540 -1 Falling back to public interface 2024-09-18T11:43:04.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:05.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:05.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:05.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:43:05.322 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:43:05.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:05.322 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:43:05.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:06.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:06.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:43:06.753 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:43:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:06.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:43:07.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:07.647 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:07.641+0000 7f85401d2540 -1 osd.2 36 log_to_monitors true 2024-09-18T11:43:08.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:08.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:08.170 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:43:08.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:43:08.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:08.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:43:08.764 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 46 up_thru 19 down_at 37 last_clean_interval [19,36) [v2:127.0.0.1:6818/272066,v1:127.0.0.1:6819/272066] [v2:127.0.0.1:6820/272066,v1:127.0.0.1:6821/272066] exists,up af5f3a42-ad46-42b6-b32b-77a5283e5652 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:43:08.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:43:08.766 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:43:08.766 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:43:08.766 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:43:08.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:43:08.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:43:08.767 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:43:08.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:43:08.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:43:08.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:43:08.987 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:43:08.987 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:43:09.633 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:43:09.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803815 2024-09-18T11:43:09.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803815 2024-09-18T11:43:09.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815' 2024-09-18T11:43:09.915 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:43:09.915 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:43:10.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574877 2024-09-18T11:43:10.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574877 2024-09-18T11:43:10.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-55834574877' 2024-09-18T11:43:10.072 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:43:10.072 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:43:10.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495619 2024-09-18T11:43:10.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495619 2024-09-18T11:43:10.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-55834574877 2-197568495619' 2024-09-18T11:43:10.294 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:43:10.294 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:43:10.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084308 2024-09-18T11:43:10.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084308 2024-09-18T11:43:10.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-55834574877 2-197568495619 3-120259084308' 2024-09-18T11:43:10.455 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:43:10.455 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803815 2024-09-18T11:43:10.456 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:43:10.458 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:43:10.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803815 2024-09-18T11:43:10.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:43:10.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803815 2024-09-18T11:43:10.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-18T11:43:10.462 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803815 2024-09-18T11:43:10.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:43:10.924 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803815 -lt 25769803815 2024-09-18T11:43:10.925 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:43:10.926 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574877 2024-09-18T11:43:10.926 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:43:10.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:43:10.929 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574877 2024-09-18T11:43:10.929 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:43:10.931 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574877 2024-09-18T11:43:10.932 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574877' 2024-09-18T11:43:10.932 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574877 2024-09-18T11:43:10.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:43:11.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574877 -lt 55834574877 2024-09-18T11:43:11.334 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:43:11.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-197568495619 2024-09-18T11:43:11.335 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:43:11.337 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:43:11.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-197568495619 2024-09-18T11:43:11.338 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:43:11.340 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495619 2024-09-18T11:43:11.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-09-18T11:43:11.341 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 197568495619 2024-09-18T11:43:11.341 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:43:11.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-18T11:43:11.768 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:43:11.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-120259084308 2024-09-18T11:43:11.769 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:43:11.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:43:11.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-120259084308 2024-09-18T11:43:11.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:43:11.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084308 2024-09-18T11:43:11.776 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 120259084308' 2024-09-18T11:43:11.776 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 120259084308 2024-09-18T11:43:11.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:43:12.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084308 -lt 120259084308 2024-09-18T11:43:12.169 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:43:12.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:12.170 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:12.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:43:12.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:43:12.715 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:43:12.715 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:43:12.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:43:12.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:43:12.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:43:12.716 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:43:13.153 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:43:13.154 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:43:13.154 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:13.155 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:13.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:43:13.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-09-18T11:43:13.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-18T11:43:13.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-09-18T11:43:13.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:43:13.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:43:13.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:43:13.772 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:43:13.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:43:13.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:43:13.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:43:13.773 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:43:13.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:43:13.774 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:43:14.163 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:43:14.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:43:14.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:14.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:14.694 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:43:14.695 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:43:14.696 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:43:15.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=682 2024-09-18T11:43:15.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 682 '!=' null 2024-09-18T11:43:15.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T11:43:15.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:43:15.219 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:43:15.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:43:15.321 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:43:15.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:43:15.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:43:15.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:43:15.322 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:43:15.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:43:15.323 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:43:15.751 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-18T11:43:15.752 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:43:15.752 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:15.752 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-09-18T11:43:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-09-18T11:43:16.250 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-18T11:43:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-18T11:43:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-18T11:43:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-18T11:43:16.251 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-18T11:43:16.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-18T11:43:16.251 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-18T11:43:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=525 2024-09-18T11:43:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 525 '!=' null 2024-09-18T11:43:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-18T11:43:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-18T11:43:16.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-18T11:43:16.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-18T11:43:16.906 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:43:16.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:43:16.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:43:16.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:43:16.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:43:16.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:43:16.909 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:43:17.299 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:43:17.300 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:43:17.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:43:17.301 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:43:17.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:43:17.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:43:17.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:43:17.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:43:17.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:43:17.824 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:43:18.368 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:43:18.376 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:43:18.958 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:43:18.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:43:18.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:43:18.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:43:18.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:43:18.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:43:18.968 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:43:18.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:43:18.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:43:18.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:43:19.085 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:43:19.085 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:43:19.087 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:43:19.088 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:43:19.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:43:19.090 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:43:19.090 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:43:19.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:19.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:43:19.092 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:43:19.093 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:19.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:43:19.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:43:19.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:43:19.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:43:19.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.120 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:19.120 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:43:19.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:43:19.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:43:19.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:43:19.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:43:19.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:43:19.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:43:19.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:43:19.128 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:43:19.129 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:43:19.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:43:19.131 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:43:19.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:43:19.133 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:43:19.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:43:19.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:19.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:43:19.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:43:19.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:43:19.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:43:19.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:43:19.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:43:19.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:43:19.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.142 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:19.142 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:43:19.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:43:19.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:43:19.143 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-09-18T11:43:19.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-18T11:43:19.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:19.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.79554 2024-09-18T11:43:19.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-18T11:43:19.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-18T11:43:19.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-09-18T11:43:19.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-09-18T11:43:19.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-18T11:43:19.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:19.511 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:19.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:19.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.512 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:19.512 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:19.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-18T11:43:19.849 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-18T11:43:19.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-18T11:43:19.851 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:43:19.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:43:19.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-18T11:43:19.852 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-18T11:43:19.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:43:19.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:43:19.853 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:43:19.854 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:43:19.854 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.854 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:19.854 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:43:19.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:43:19.855 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get fsid 2024-09-18T11:43:19.946 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-18T11:43:19.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-18T11:43:19.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-18T11:43:19.947 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:19.948 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:19.949 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:43:19.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-18T11:43:19.949 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.79554/ceph-mon.a.asok config get mon_host 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-09-18T11:43:20.035 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-18T11:43:20.931 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-18T11:43:20.931 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:20.931 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:20.931 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:20.931 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:20.932 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:20.932 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:20.933 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:43:20.934 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-18T11:43:20.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-09-18T11:43:20.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-18T11:43:21.291 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'rbd' already exists 2024-09-18T11:43:21.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:43:22.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-09-18T11:43:22.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-18T11:43:22.308 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-18T11:43:22.309 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:43:22.309 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:22.309 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:22.309 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-mon.a.asok 2024-09-18T11:43:22.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-09-18T11:43:22.310 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-mon.a.asok log flush 2024-09-18T11:43:22.386 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:43:22.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-09-18T11:43:22.397 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:43:19.549+0000 7fe2d3ce6a00 0 load: jerasure load: lrc 2024-09-18T11:43:22.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-09-18T11:43:22.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-09-18T11:43:22.398 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-09-18T11:43:22.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:347: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-09-18T11:43:22.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-09-18T11:43:22.399 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-09-18T11:43:22.399 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-09-18T11:43:22.401 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:43:22.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:22.405 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:43:22.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:43:22.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:43:22.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-09-18T11:43:22.408 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:43:22.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c54a804-212b-4041-9d5e-75247806d8dd 2024-09-18T11:43:22.408 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6c54a804-212b-4041-9d5e-75247806d8dd' 2024-09-18T11:43:22.409 INFO:tasks.workunit.client.0.smithi174.stdout:add osd0 6c54a804-212b-4041-9d5e-75247806d8dd 2024-09-18T11:43:22.409 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:43:22.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDavOpmHrh4GRAAE7OV3xyKz8MqwXAA3aom2g== 2024-09-18T11:43:22.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDavOpmHrh4GRAAE7OV3xyKz8MqwXAA3aom2g=="}' 2024-09-18T11:43:22.431 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c54a804-212b-4041-9d5e-75247806d8dd -i td/test-erasure-eio/0/new.json 2024-09-18T11:43:22.770 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:43:22.785 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-09-18T11:43:22.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDavOpmHrh4GRAAE7OV3xyKz8MqwXAA3aom2g== --osd-uuid 6c54a804-212b-4041-9d5e-75247806d8dd 2024-09-18T11:43:22.819 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.813+0000 7f7606891540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:22.822 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.817+0000 7f7606891540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:22.824 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.817+0000 7f7606891540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:22.824 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.817+0000 7f7606891540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:43:22.824 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.817+0000 7f7606891540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:43:22.824 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.817+0000 7f7606891540 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-09-18T11:43:22.951 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:22.945+0000 7f7606891540 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-09-18T11:43:29.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-09-18T11:43:29.019 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:43:29.021 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd0 key to auth repository 2024-09-18T11:43:29.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-18T11:43:29.022 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:43:29.591 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.0 2024-09-18T11:43:29.591 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-18T11:43:29.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:43:29.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:43:29.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:43:29.592 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:43:29.621 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:29.613+0000 7fbe6be99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:29.624 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:29.617+0000 7fbe6be99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:29.630 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:29.617+0000 7fbe6be99540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:43:29.989 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:43:29.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:29.990 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:30.160 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:30.153+0000 7fbe6be99540 -1 Falling back to public interface 2024-09-18T11:43:30.443 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:31.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:43:31.446 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:43:31.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:31.446 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:31.841 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:32.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:32.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:32.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:43:32.843 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:43:32.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:32.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:32.958 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:32.953+0000 7fbe6be99540 -1 osd.0 0 log_to_monitors true 2024-09-18T11:43:33.325 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:34.327 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:34.328 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:43:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:43:34.328 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:34.329 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:34.762 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:35.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:35.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:35.765 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:43:35.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:43:35.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:35.765 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:36.187 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:37.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:37.189 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:37.190 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:43:37.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:43:37.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:37.190 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:37.668 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:38.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:38.670 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:38.670 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:43:38.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:43:38.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:38.671 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:39.095 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:40.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:40.098 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:40.098 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:43:40.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:43:40.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:40.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:40.662 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:41.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:41.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:41.664 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:43:41.665 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:43:41.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:41.665 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:42.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:43.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:43.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:43.102 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:43:43.102 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:43:43.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:43.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:43.696 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:44.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:44.698 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:44.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:43:44.699 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:43:44.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:44.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:45.168 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:46.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:46.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:46.170 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:43:46.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:43:46.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:46.171 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:46.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:47.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:47.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:47.742 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:43:47.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:43:47.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:47.743 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:48.515 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:49.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:49.517 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:49.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:43:49.518 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:43:49.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:49.518 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:50.742 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:51.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:51.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:51.745 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:43:51.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:43:51.745 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:51.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:52.744 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:53.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:53.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:53.746 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:43:53.747 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:43:53.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:53.747 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:54.521 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:55.522 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:55.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:55.523 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:43:55.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:43:55.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:55.523 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:56.330 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:43:57.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:43:57.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:43:57.332 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:43:57.333 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:43:57.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:43:57.333 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-18T11:43:57.752 INFO:tasks.workunit.client.0.smithi174.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/274502,v1:127.0.0.1:6803/274502] [v2:127.0.0.1:6804/274502,v1:127.0.0.1:6805/274502] exists,up 6c54a804-212b-4041-9d5e-75247806d8dd 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:43:57.754 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:43:57.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:43:57.756 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:43:57.756 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:43:57.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-09-18T11:43:57.759 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:43:57.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9717c04a-4b3f-4154-b45e-3b7b079da823 2024-09-18T11:43:57.760 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9717c04a-4b3f-4154-b45e-3b7b079da823' 2024-09-18T11:43:57.760 INFO:tasks.workunit.client.0.smithi174.stdout:add osd1 9717c04a-4b3f-4154-b45e-3b7b079da823 2024-09-18T11:43:57.761 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:43:57.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9vOpmDNwyLhAA4zVBd4iTpKOZ4FtFQbKl0g== 2024-09-18T11:43:57.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9vOpmDNwyLhAA4zVBd4iTpKOZ4FtFQbKl0g=="}' 2024-09-18T11:43:57.778 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9717c04a-4b3f-4154-b45e-3b7b079da823 -i td/test-erasure-eio/1/new.json 2024-09-18T11:43:58.377 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:43:58.394 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-09-18T11:43:58.396 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9vOpmDNwyLhAA4zVBd4iTpKOZ4FtFQbKl0g== --osd-uuid 9717c04a-4b3f-4154-b45e-3b7b079da823 2024-09-18T11:43:58.429 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.421+0000 7f568ea42540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:58.431 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.425+0000 7f568ea42540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:58.433 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.425+0000 7f568ea42540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:43:58.434 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.429+0000 7f568ea42540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:43:58.434 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.429+0000 7f568ea42540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:43:58.434 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.429+0000 7f568ea42540 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-09-18T11:43:58.519 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:43:58.513+0000 7f568ea42540 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-09-18T11:44:04.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-09-18T11:44:04.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:44:04.805 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd1 key to auth repository 2024-09-18T11:44:04.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-18T11:44:04.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:44:05.337 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.1 2024-09-18T11:44:05.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-18T11:44:05.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:44:05.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:44:05.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:44:05.338 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:44:05.367 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:05.361+0000 7fec53a0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:05.370 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:05.365+0000 7fec53a0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:05.372 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:05.365+0000 7fec53a0c540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:05.714 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:44:05.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:44:05.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:05.715 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:05.908 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:05.901+0000 7fec53a0c540 -1 Falling back to public interface 2024-09-18T11:44:06.119 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:07.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:07.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:07.121 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:44:07.122 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:44:07.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:07.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:07.542 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:08.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:08.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:08.544 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:44:08.544 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:44:08.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:08.545 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:08.772 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:08.769+0000 7fec53a0c540 -1 osd.1 0 log_to_monitors true 2024-09-18T11:44:08.996 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:09.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:09.998 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:09.998 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:44:09.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:44:09.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:09.999 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:10.415 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:11.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:11.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:11.417 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:44:11.417 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:44:11.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:11.418 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:11.926 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:12.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:12.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:12.928 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:44:12.928 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:44:12.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:12.929 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:13.504 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:14.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:14.506 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:14.506 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:44:14.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:44:14.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:14.507 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:14.941 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:15.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:15.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:15.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:44:15.944 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:44:15.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:15.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:16.423 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:17.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:17.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:17.425 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:44:17.426 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:44:17.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:17.426 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:17.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:18.901 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:18.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:18.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:44:18.902 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:44:18.902 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:18.903 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:19.445 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:20.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:20.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:20.447 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:44:20.447 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:44:20.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:20.448 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:20.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:21.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:21.969 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:21.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:44:21.970 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:44:21.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:21.970 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:22.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:23.469 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:23.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:23.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:44:23.470 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:44:23.470 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:23.471 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:24.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:25.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:25.553 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:25.553 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:44:25.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:44:25.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:25.554 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:26.548 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:27.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:27.550 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:27.550 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:44:27.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:44:27.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:27.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:28.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:29.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:29.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:29.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:44:29.060 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:44:29.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:29.060 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:30.283 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:31.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:31.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:31.285 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:44:31.285 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:44:31.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:31.286 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:31.830 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:32.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:32.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:32.833 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:44:32.833 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:44:32.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:32.834 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:33.634 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:34.636 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:34.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:34.637 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:44:34.637 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:44:34.639 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:34.640 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-18T11:44:35.142 INFO:tasks.workunit.client.0.smithi174.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/276277,v1:127.0.0.1:6811/276277] [v2:127.0.0.1:6812/276277,v1:127.0.0.1:6813/276277] exists,up 9717c04a-4b3f-4154-b45e-3b7b079da823 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:44:35.144 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:44:35.145 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:44:35.146 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:44:35.148 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:44:35.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:44:35.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:44:35.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:44:35.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:44:35.149 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:44:35.149 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1 2024-09-18T11:44:35.150 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1' 2024-09-18T11:44:35.150 INFO:tasks.workunit.client.0.smithi174.stdout:add osd2 c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1 2024-09-18T11:44:35.150 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:44:35.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjvepmHjb8CRAAGmckyspExQLk5cs5vuuj+A== 2024-09-18T11:44:35.169 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjvepmHjb8CRAAGmckyspExQLk5cs5vuuj+A=="}' 2024-09-18T11:44:35.170 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1 -i td/test-erasure-eio/2/new.json 2024-09-18T11:44:35.926 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:44:35.943 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-09-18T11:44:35.944 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjvepmHjb8CRAAGmckyspExQLk5cs5vuuj+A== --osd-uuid c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1 2024-09-18T11:44:35.978 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:35.972+0000 7f4bdc81a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:35.982 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:35.980+0000 7f4bdc81a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:35.983 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:35.980+0000 7f4bdc81a540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:35.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:35.980+0000 7f4bdc81a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:44:35.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:35.980+0000 7f4bdc81a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:44:35.985 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:35.980+0000 7f4bdc81a540 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-09-18T11:44:36.060 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:36.056+0000 7f4bdc81a540 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-09-18T11:44:42.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-09-18T11:44:42.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:44:42.621 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd2 key to auth repository 2024-09-18T11:44:42.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-18T11:44:42.621 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:44:43.158 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-18T11:44:43.158 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:44:43.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:44:43.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:44:43.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:44:43.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:44:43.187 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:43.184+0000 7f9103424540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:43.189 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:43.184+0000 7f9103424540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:43.190 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:43.188+0000 7f9103424540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:43.613 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:44:43.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:44:43.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:43.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:43.752 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:43.748+0000 7f9103424540 -1 Falling back to public interface 2024-09-18T11:44:44.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:45.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:45.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:45.033 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:44:45.034 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:44:45.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:45.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:45.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:46.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:46.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:46.513 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:44:46.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:44:46.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:46.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:46.530 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:44:46.528+0000 7f9103424540 -1 osd.2 0 log_to_monitors true 2024-09-18T11:44:47.051 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:48.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:48.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:48.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:44:48.053 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:44:48.053 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:48.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:48.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:49.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:49.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:49.616 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:44:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:44:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:49.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:50.107 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:51.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:51.109 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:51.109 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:44:51.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:44:51.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:51.110 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:51.805 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:52.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:52.808 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:52.809 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:44:52.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:44:52.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:52.809 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:53.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:54.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:54.320 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:44:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:44:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:54.320 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:54.800 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:55.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:55.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:55.803 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:44:55.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:44:55.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:55.804 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:56.427 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:57.429 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:57.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:57.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:44:57.430 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:44:57.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:57.430 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:57.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:44:58.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:44:58.898 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:44:58.899 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:44:58.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:44:58.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:44:58.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:44:59.645 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:00.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:00.647 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:00.648 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:45:00.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:45:00.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:00.648 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:01.257 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:02.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:02.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:02.259 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:45:02.259 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:45:02.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:02.260 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:03.375 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:04.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:04.377 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:04.378 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:45:04.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:45:04.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:04.378 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:05.136 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:06.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:06.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:06.139 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:45:06.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:45:06.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:06.139 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:06.965 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:07.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:07.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:07.967 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:45:07.968 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:45:07.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:07.968 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:08.750 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:09.752 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:09.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:09.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:45:09.753 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:45:09.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:09.753 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:10.559 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:11.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:11.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:11.561 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:45:11.562 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:45:11.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:11.562 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:45:12.128 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/278078,v1:127.0.0.1:6819/278078] [v2:127.0.0.1:6820/278078,v1:127.0.0.1:6821/278078] exists,up c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1 2024-09-18T11:45:12.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:45:12.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:45:12.132 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-18T11:45:12.134 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-18T11:45:12.135 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-09-18T11:45:12.137 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-18T11:45:12.137 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3c11e919-1835-4aeb-9ee5-50557f2a7130 2024-09-18T11:45:12.137 INFO:tasks.workunit.client.0.smithi174.stdout:add osd3 3c11e919-1835-4aeb-9ee5-50557f2a7130 2024-09-18T11:45:12.138 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3c11e919-1835-4aeb-9ee5-50557f2a7130' 2024-09-18T11:45:12.138 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-18T11:45:12.159 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIvepmYodnCRAAH9BrqcubDpBhJEu/oynpNw== 2024-09-18T11:45:12.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIvepmYodnCRAAH9BrqcubDpBhJEu/oynpNw=="}' 2024-09-18T11:45:12.160 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3c11e919-1835-4aeb-9ee5-50557f2a7130 -i td/test-erasure-eio/3/new.json 2024-09-18T11:45:13.192 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:45:13.204 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-09-18T11:45:13.206 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIvepmYodnCRAAH9BrqcubDpBhJEu/oynpNw== --osd-uuid 3c11e919-1835-4aeb-9ee5-50557f2a7130 2024-09-18T11:45:13.238 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.236+0000 7f6d59160540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:13.252 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.248+0000 7f6d59160540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:13.253 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.248+0000 7f6d59160540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:13.255 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.252+0000 7f6d59160540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:45:13.255 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.252+0000 7f6d59160540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:45:13.255 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.252+0000 7f6d59160540 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-09-18T11:45:13.299 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:13.296+0000 7f6d59160540 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-09-18T11:45:19.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-09-18T11:45:19.795 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-18T11:45:19.797 INFO:tasks.workunit.client.0.smithi174.stdout:adding osd3 key to auth repository 2024-09-18T11:45:19.797 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-18T11:45:19.798 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-18T11:45:20.360 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.3 2024-09-18T11:45:20.360 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-18T11:45:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:45:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-18T11:45:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-18T11:45:20.361 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-18T11:45:20.391 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:20.388+0000 7f868ade7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:20.415 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:20.412+0000 7f868ade7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:20.416 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:20.412+0000 7f868ade7540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:45:20.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-18T11:45:20.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:45:20.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-18T11:45:20.802 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:45:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:45:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:45:20.803 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:45:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:20.803 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:20.980 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:20.976+0000 7f868ade7540 -1 Falling back to public interface 2024-09-18T11:45:21.195 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:22.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:22.197 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:22.198 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:45:22.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:45:22.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:22.198 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:22.611 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:23.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:23.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:23.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:45:23.613 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:45:23.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:23.614 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:23.915 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:45:23.911+0000 7f868ade7540 -1 osd.3 0 log_to_monitors true 2024-09-18T11:45:24.016 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:25.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:25.020 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:25.020 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:45:25.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:45:25.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:25.021 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:25.511 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:26.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:26.513 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:26.514 INFO:tasks.workunit.client.0.smithi174.stdout:4 2024-09-18T11:45:26.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-18T11:45:26.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:26.514 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:27.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:28.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:28.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:28.131 INFO:tasks.workunit.client.0.smithi174.stdout:5 2024-09-18T11:45:28.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-18T11:45:28.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:28.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:28.583 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:29.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:29.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:29.585 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-18T11:45:29.586 INFO:tasks.workunit.client.0.smithi174.stdout:6 2024-09-18T11:45:29.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:29.586 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:30.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:31.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:31.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:31.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-18T11:45:31.044 INFO:tasks.workunit.client.0.smithi174.stdout:7 2024-09-18T11:45:31.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:31.044 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:31.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:32.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:32.569 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:32.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-18T11:45:32.570 INFO:tasks.workunit.client.0.smithi174.stdout:8 2024-09-18T11:45:32.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:32.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:33.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:34.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:34.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:34.106 INFO:tasks.workunit.client.0.smithi174.stdout:9 2024-09-18T11:45:34.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-18T11:45:34.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:34.106 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:34.613 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:35.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:35.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:35.615 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-18T11:45:35.615 INFO:tasks.workunit.client.0.smithi174.stdout:10 2024-09-18T11:45:35.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:35.616 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:36.122 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:37.124 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:37.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:37.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-18T11:45:37.125 INFO:tasks.workunit.client.0.smithi174.stdout:11 2024-09-18T11:45:37.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:37.125 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:37.564 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:38.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:38.567 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:38.567 INFO:tasks.workunit.client.0.smithi174.stdout:12 2024-09-18T11:45:38.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-18T11:45:38.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:38.568 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:39.129 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:40.130 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:40.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:40.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-18T11:45:40.131 INFO:tasks.workunit.client.0.smithi174.stdout:13 2024-09-18T11:45:40.131 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:40.132 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:40.890 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:41.891 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:41.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:41.892 INFO:tasks.workunit.client.0.smithi174.stdout:14 2024-09-18T11:45:41.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-18T11:45:41.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:41.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:42.856 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:43.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:43.861 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:43.861 INFO:tasks.workunit.client.0.smithi174.stdout:15 2024-09-18T11:45:43.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-18T11:45:43.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:43.862 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:44.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:45.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:45.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:45.707 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-18T11:45:45.707 INFO:tasks.workunit.client.0.smithi174.stdout:16 2024-09-18T11:45:45.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:45.708 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:46.599 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:47.601 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-18T11:45:47.601 INFO:tasks.workunit.client.0.smithi174.stdout:17 2024-09-18T11:45:47.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:47.602 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:48.315 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:45:49.317 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:45:49.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:45:49.318 INFO:tasks.workunit.client.0.smithi174.stdout:18 2024-09-18T11:45:49.318 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-18T11:45:49.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:45:49.319 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-18T11:45:49.786 INFO:tasks.workunit.client.0.smithi174.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/279857,v1:127.0.0.1:6827/279857] [v2:127.0.0.1:6828/279857,v1:127.0.0.1:6829/279857] exists,up 3c11e919-1835-4aeb-9ee5-50557f2a7130 2024-09-18T11:45:49.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:45:49.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:45:49.787 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:45:49.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-09-18T11:45:49.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-18T11:45:49.788 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-18T11:45:49.788 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:45:49.789 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:45:49.789 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:45:49.789 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.0.asok 2024-09-18T11:45:49.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-09-18T11:45:49.790 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.0.asok log flush 2024-09-18T11:45:49.877 INFO:tasks.workunit.client.0.smithi174.stdout:{} 2024-09-18T11:45:49.884 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-09-18T11:45:49.891 INFO:tasks.workunit.client.0.smithi174.stdout:2024-09-18T11:43:30.161+0000 7fbe6be99540 0 load: jerasure load: lrc 2024-09-18T11:45:49.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:349: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-09-18T11:45:49.893 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-09-18T11:45:50.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-09-18T11:45:50.739 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-09-18T11:45:52.447 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' already exists 2024-09-18T11:45:52.456 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-18T11:45:53.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-09-18T11:45:53.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:45:53.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:45:53.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:45:53.459 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:45:53.459 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:45:53.461 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:45:53.461 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:45:53.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:45:53.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:45:53.461 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:45:53.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:45:53.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:45:53.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:45:53.689 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:45:53.689 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:45:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:45:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:45:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:45:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:45:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:45:54.118 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:45:54.119 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:45:54.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803811 2024-09-18T11:45:54.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803811 2024-09-18T11:45:54.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811' 2024-09-18T11:45:54.266 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:45:54.266 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:45:54.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574874 2024-09-18T11:45:54.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574874 2024-09-18T11:45:54.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574874' 2024-09-18T11:45:54.407 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:45:54.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:45:54.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378646 2024-09-18T11:45:54.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378646 2024-09-18T11:45:54.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574874 2-81604378646' 2024-09-18T11:45:54.551 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:45:54.551 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:45:54.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051597 2024-09-18T11:45:54.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051597 2024-09-18T11:45:54.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574874 2-81604378646 3-124554051597' 2024-09-18T11:45:54.699 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:45:54.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803811 2024-09-18T11:45:54.700 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:45:54.702 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:45:54.703 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803811 2024-09-18T11:45:54.704 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:45:54.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803811 2024-09-18T11:45:54.705 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-18T11:45:54.705 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803811 2024-09-18T11:45:54.706 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:45:55.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-18T11:45:55.104 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:45:56.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:45:56.106 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:45:56.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-18T11:45:56.566 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:45:56.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574874 2024-09-18T11:45:56.568 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:45:56.570 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:45:56.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574874 2024-09-18T11:45:56.571 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:45:56.573 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574874 2024-09-18T11:45:56.574 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574874 2024-09-18T11:45:56.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-09-18T11:45:56.575 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:45:57.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574875 -lt 55834574874 2024-09-18T11:45:57.028 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:45:57.029 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378646 2024-09-18T11:45:57.029 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:45:57.031 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:45:57.032 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378646 2024-09-18T11:45:57.033 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:45:57.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378646 2024-09-18T11:45:57.034 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378646' 2024-09-18T11:45:57.034 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 81604378646 2024-09-18T11:45:57.035 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:45:57.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378646 -lt 81604378646 2024-09-18T11:45:57.462 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:45:57.462 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051597 2024-09-18T11:45:57.463 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:45:57.464 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:45:57.466 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051597 2024-09-18T11:45:57.467 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:45:57.467 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051597 2024-09-18T11:45:57.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051597 2024-09-18T11:45:57.467 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051597' 2024-09-18T11:45:57.468 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:45:57.877 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051598 -lt 124554051597 2024-09-18T11:45:57.878 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:45:57.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:45:57.879 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:45:58.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:45:58.397 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:45:58.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:45:58.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:45:58.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:45:58.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:45:58.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:45:58.398 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:45:58.791 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:45:58.792 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:45:58.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:45:58.793 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:45:59.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:45:59.341 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:352: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-79554-1 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-79554-1 2024-09-18T11:45:59.342 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-79554-1 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-09-18T11:45:59.343 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-79554-1 td/test-erasure-eio/ORIGINAL 2024-09-18T11:45:59.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-79554-1 td/test-erasure-eio 1 2024-09-18T11:45:59.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=ec 2024-09-18T11:45:59.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-18T11:45:59.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-18T11:45:59.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-jerasure 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-79554-1 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/test-erasure-eio 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-18T11:45:59.525 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:45:59.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-jerasure obj-eio-79554-1 2024-09-18T11:45:59.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:45:59.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-1 2024-09-18T11:45:59.526 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-1 2024-09-18T11:45:59.526 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:46:00.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:46:00.056 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:46:00.056 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:46:00.056 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:46:00.056 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-18T11:46:00.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-18T11:46:00.057 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' ec '!=' ec ']' 2024-09-18T11:46:00.057 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/test-erasure-eio/1/type 2024-09-18T11:46:00.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-18T11:46:00.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-18T11:46:00.058 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-18T11:46:00.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-18T11:46:00.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-18T11:46:00.059 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-18T11:46:00.060 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-18T11:46:00.060 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-18T11:46:00.060 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:46:00.060 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:46:00.061 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:46:00.061 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:00.061 INFO:tasks.workunit.client.0.smithi174.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:46:00.062 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:46:00.062 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.79554/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-18T11:46:00.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-18T11:46:00.161 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-18T11:46:00.162 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-18T11:46:00.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-18T11:46:00.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-18T11:46:00.163 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-18T11:46:00.164 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-18T11:46:00.164 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:00.164 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:46:00.164 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.79554/ceph-osd.1.asok 2024-09-18T11:46:00.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-18T11:46:00.165 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.79554/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-79554-1 1 2024-09-18T11:46:00.253 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-79554-1 2024-09-18T11:46:00.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-09-18T11:46:00.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-09-18T11:46:00.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-79554-1 2024-09-18T11:46:00.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-09-18T11:46:00.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-09-18T11:46:00.254 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-79554-1 td/test-erasure-eio/COPY 2024-09-18T11:46:00.300 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-09-18T11:46:00.301 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-09-18T11:46:00.302 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-09-18T11:46:00.303 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=($(get_osds $poolname $objname)) 2024-09-18T11:46:00.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-79554-1 2024-09-18T11:46:00.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:46:00.303 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-1 2024-09-18T11:46:00.305 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-1 2024-09-18T11:46:00.305 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:46:00.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:46:00.755 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:46:00.755 INFO:tasks.workunit.client.0.smithi174.stderr:2' 2024-09-18T11:46:00.755 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-09-18T11:46:00.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-09-18T11:46:00.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-09-18T11:46:00.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-09-18T11:46:00.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:00.757 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:00.757 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:00.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:00.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:01.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:01.071 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-09-18T11:46:02.227 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already out. 2024-09-18T11:46:02.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-79554-1 2024-09-18T11:46:02.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-09-18T11:46:02.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-09-18T11:46:02.240 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-79554-1 2024-09-18T11:46:02.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-79554-1 2024-09-18T11:46:02.241 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-18T11:46:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-18T11:46:02.680 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:46:02.681 INFO:tasks.workunit.client.0.smithi174.stderr:0' 2024-09-18T11:46:02.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-18T11:46:02.681 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-09-18T11:46:03.565 INFO:tasks.workunit.client.0.smithi174.stderr:osd.2 is already in. 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-09-18T11:46:03.574 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-18T11:46:03.575 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-09-18T11:46:03.576 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-18T11:46:03.577 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-18T11:46:03.578 INFO:tasks.workunit.client.0.smithi174.stdout:start osd.2 2024-09-18T11:46:03.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1a1a6a19-c53d-42e3-ad37-af488a0f522e --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.79554/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-18T11:46:03.579 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-09-18T11:46:03.579 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-18T11:46:03.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-18T11:46:03.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-18T11:46:03.580 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-18T11:46:03.610 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:46:03.607+0000 7f81646da540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:03.706 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:46:03.699+0000 7f81646da540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:03.709 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:46:03.703+0000 7f81646da540 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-18T11:46:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-18T11:46:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-18T11:46:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-18T11:46:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-18T11:46:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-18T11:46:04.225 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:04.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-18T11:46:04.226 INFO:tasks.workunit.client.0.smithi174.stdout:0 2024-09-18T11:46:04.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:04.226 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:46:04.269 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:46:04.263+0000 7f81646da540 -1 Falling back to public interface 2024-09-18T11:46:04.843 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:05.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:05.844 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:05.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-18T11:46:05.845 INFO:tasks.workunit.client.0.smithi174.stdout:1 2024-09-18T11:46:05.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:05.845 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:46:06.261 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:07.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:07.263 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:07.264 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-18T11:46:07.264 INFO:tasks.workunit.client.0.smithi174.stdout:2 2024-09-18T11:46:07.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:07.265 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:46:07.536 INFO:tasks.workunit.client.0.smithi174.stderr:2024-09-18T11:46:07.531+0000 7f81646da540 -1 osd.2 36 log_to_monitors true 2024-09-18T11:46:07.756 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-18T11:46:08.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-18T11:46:08.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-18T11:46:08.758 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-18T11:46:08.758 INFO:tasks.workunit.client.0.smithi174.stdout:3 2024-09-18T11:46:08.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-18T11:46:08.759 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-18T11:46:09.523 INFO:tasks.workunit.client.0.smithi174.stdout:osd.2 up in weight 1 up_from 46 up_thru 19 down_at 37 last_clean_interval [19,36) [v2:127.0.0.1:6818/281546,v1:127.0.0.1:6819/281546] [v2:127.0.0.1:6820/281546,v1:127.0.0.1:6821/281546] exists,up c4b3fd01-2d4a-4e6a-acc8-f62f2864ebe1 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-18T11:46:09.524 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2024-09-18T11:46:09.525 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-18T11:46:09.525 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-18T11:46:09.525 INFO:tasks.workunit.client.0.smithi174.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-18T11:46:09.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-18T11:46:09.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-18T11:46:09.526 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-18T11:46:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-18T11:46:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-18T11:46:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-18T11:46:09.740 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-18T11:46:09.741 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-18T11:46:10.404 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-18T11:46:10.404 INFO:tasks.workunit.client.0.smithi174.stderr:1 2024-09-18T11:46:10.404 INFO:tasks.workunit.client.0.smithi174.stderr:2 2024-09-18T11:46:10.405 INFO:tasks.workunit.client.0.smithi174.stderr:3' 2024-09-18T11:46:10.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-18T11:46:10.405 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:46:10.405 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-18T11:46:10.619 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803816 2024-09-18T11:46:10.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803816 2024-09-18T11:46:10.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816' 2024-09-18T11:46:10.620 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:46:10.620 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-18T11:46:10.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574880 2024-09-18T11:46:10.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574880 2024-09-18T11:46:10.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574880' 2024-09-18T11:46:10.777 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:46:10.777 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-18T11:46:10.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=197568495619 2024-09-18T11:46:10.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 197568495619 2024-09-18T11:46:10.917 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574880 2-197568495619' 2024-09-18T11:46:10.918 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-18T11:46:10.918 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-18T11:46:11.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=124554051603 2024-09-18T11:46:11.061 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 124554051603 2024-09-18T11:46:11.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803816 1-55834574880 2-197568495619 3-124554051603' 2024-09-18T11:46:11.062 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:46:11.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803816 2024-09-18T11:46:11.063 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:46:11.064 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-18T11:46:11.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803816 2024-09-18T11:46:11.066 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:46:11.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803816 2024-09-18T11:46:11.068 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803816' 2024-09-18T11:46:11.068 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.0 seq 25769803816 2024-09-18T11:46:11.069 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:46:11.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803814 -lt 25769803816 2024-09-18T11:46:11.484 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-18T11:46:12.486 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-18T11:46:12.487 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-18T11:46:12.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803816 -lt 25769803816 2024-09-18T11:46:12.892 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:46:12.896 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574880 2024-09-18T11:46:12.896 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:46:12.896 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-18T11:46:12.896 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574880 2024-09-18T11:46:12.897 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:46:12.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574880 2024-09-18T11:46:12.899 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.1 seq 55834574880 2024-09-18T11:46:12.899 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2024-09-18T11:46:12.900 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-18T11:46:13.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574880 -lt 55834574880 2024-09-18T11:46:13.406 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:46:13.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-197568495619 2024-09-18T11:46:13.407 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:46:13.410 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-18T11:46:13.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-197568495619 2024-09-18T11:46:13.411 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:46:13.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=197568495619 2024-09-18T11:46:13.413 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-09-18T11:46:13.413 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.2 seq 197568495619 2024-09-18T11:46:13.413 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-18T11:46:13.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 197568495619 -lt 197568495619 2024-09-18T11:46:13.828 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-18T11:46:13.829 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-124554051603 2024-09-18T11:46:13.830 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-18T11:46:13.832 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-18T11:46:13.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-124554051603 2024-09-18T11:46:13.833 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-18T11:46:13.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=124554051603 2024-09-18T11:46:13.835 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 124554051603' 2024-09-18T11:46:13.835 INFO:tasks.workunit.client.0.smithi174.stdout:waiting osd.3 seq 124554051603 2024-09-18T11:46:13.836 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-18T11:46:14.215 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 124554051603 -lt 124554051603 2024-09-18T11:46:14.215 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-18T11:46:14.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:14.216 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:14.763 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-09-18T11:46:14.764 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-18T11:46:14.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-18T11:46:14.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-18T11:46:14.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-18T11:46:14.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-18T11:46:14.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-18T11:46:14.765 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-18T11:46:15.123 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-18T11:46:15.123 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-18T11:46:15.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-18T11:46:15.124 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-18T11:46:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-09-18T11:46:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-18T11:46:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-18T11:46:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:354: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-09-18T11:46:15.658 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-09-18T11:46:15.659 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-09-18T11:46:16.292 INFO:tasks.workunit.client.0.smithi174.stderr:pool 'pool-jerasure' does not exist 2024-09-18T11:46:16.305 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-09-18T11:46:16.909 INFO:tasks.workunit.client.0.smithi174.stderr:erasure-code-profile myprofile does not exist 2024-09-18T11:46:16.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-09-18T11:46:16.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:46:16.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-18T11:46:16.921 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:46:16.922 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:16.922 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:16.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:16.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:16.923 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:17.041 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:17.041 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:46:17.043 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:46:17.044 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:46:17.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:46:17.046 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:46:17.047 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:46:17.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:17.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:46:17.049 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:46:17.050 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:17.052 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:17.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-18T11:46:17.054 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:46:17.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:46:17.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:17.081 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:46:17.082 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/test-erasure-eio 0 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-18T11:46:17.083 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-09-18T11:46:17.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-18T11:46:17.084 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-18T11:46:17.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-18T11:46:17.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-18T11:46:17.084 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-18T11:46:17.089 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-18T11:46:17.090 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-18T11:46:17.091 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-18T11:46:17.092 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-18T11:46:17.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-18T11:46:17.093 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-18T11:46:17.094 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-18T11:46:17.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:17.096 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-18T11:46:17.097 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-18T11:46:17.097 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T11:46:17.099 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:17.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-18T11:46:17.100 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-09-18T11:46:17.102 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-18T11:46:17.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-18T11:46:17.103 INFO:tasks.workunit.client.0.smithi174.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.79554 2024-09-18T11:46:17.103 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.79554 2024-09-18T11:46:17.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-18T11:46:17.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-18T11:46:17.105 INFO:tasks.workunit.client.0.smithi174.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-18T11:46:17.106 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T11:46:17.106 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-18T11:46:17.122 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-09-18T11:46:17.122 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-18T11:46:17.431 DEBUG:teuthology.parallel:result is None 2024-09-18T11:46:17.431 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T11:46:17.446 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T11:46:17.446 DEBUG:teuthology.orchestra.run.smithi174:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-18T11:46:17.470 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-18T11:46:17.471 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T11:46:17.482 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T11:46:17.482 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T11:46:17.521 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T11:46:17.529 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T11:46:17.530 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T11:46:17.562 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T11:46:17.562 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-09-18T11:46:17.562 INFO:teuthology.orchestra.run.smithi174.stdout:+hv01.front.sepi 67.205.162.81 3 u 17 64 377 0.092 0.902 0.052 2024-09-18T11:46:17.562 INFO:teuthology.orchestra.run.smithi174.stdout:*hv02.front.sepi 63.231.80.2 3 u 9 64 377 0.104 0.949 0.207 2024-09-18T11:46:17.562 INFO:teuthology.orchestra.run.smithi174.stdout:+hv03.front.sepi 74.6.168.72 3 u 62 64 377 0.096 0.218 0.236 2024-09-18T11:46:17.562 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-18T11:46:17.563 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T11:46:17.571 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T11:46:17.572 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T11:46:17.580 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T11:46:17.588 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T11:46:17.597 INFO:teuthology.task.internal:Duration was 5201.668513 seconds 2024-09-18T11:46:17.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T11:46:17.606 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T11:46:17.606 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T11:46:17.657 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T11:46:17.657 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-09-18T11:46:17.657 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T11:46:17.669 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T11:46:17.669 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T11:46:17.753 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T11:46:17.754 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T11:46:17.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T11:46:17.849 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T11:46:17.849 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T11:46:17.864 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T11:46:17.873 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:17.919 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-09-18T11:46:17.935 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T11:46:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T11:46:17.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T11:46:17.950 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T11:46:17.952 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7910869/remote/smithi174 2024-09-18T11:46:17.953 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T11:46:17.982 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T11:46:17.982 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T11:46:18.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T11:46:18.036 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T11:46:18.036 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T11:46:18.048 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T11:46:18.049 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T11:46:18.056 INFO:teuthology.orchestra.run.smithi174.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 11:46 /home/ubuntu/cephtest 2024-09-18T11:46:18.059 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T11:46:18.072 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T11:46:18.095 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 5201.6685128211975 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T11:46:18.095 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T11:46:18.164 INFO:teuthology.run:pass